-- Logs begin at Tue 2022-10-04 13:46:51 UTC, end at Tue 2022-10-04 14:47:37 UTC. -- Oct 04 14:13:01.074536 dv-cinder-860260-1-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 04 14:13:01.083718 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 04 14:13:01.085871 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: *** Starting uWSGI 2.0.18-debian (64bit) on [Tue Oct 4 14:13:01 2022] *** Oct 04 14:13:01.085871 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 04 14:13:01.085871 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 04 14:13:01.085871 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: nodename: dv-cinder-860260-1-iscsi Oct 04 14:13:01.085871 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: machine: x86_64 Oct 04 14:13:01.085871 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: clock source: unix Oct 04 14:13:01.085871 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: pcre jit disabled Oct 04 14:13:01.085871 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: detected number of CPU cores: 4 Oct 04 14:13:01.085871 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: current working directory: / Oct 04 14:13:01.085871 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: detected binary path: /usr/bin/uwsgi-core Oct 04 14:13:01.085871 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: your processes number limit is 47680 Oct 04 14:13:01.085871 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: your memory page size is 4096 bytes Oct 04 14:13:01.086621 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: detected max file descriptor number: 2048 Oct 04 14:13:01.086621 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: lock engine: pthread robust mutexes Oct 04 14:13:01.086621 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: thunder lock: enabled Oct 04 14:13:01.086621 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 04 14:13:01.086621 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 04 14:13:01.127315 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: Python main interpreter initialized at 0x564023f029c0 Oct 04 14:13:01.127315 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: python threads support enabled Oct 04 14:13:01.127315 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: your server socket listen backlog is limited to 100 connections Oct 04 14:13:01.127315 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: your mercy for graceful operations on workers is 90 seconds Oct 04 14:13:01.127859 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: mapped 403077 bytes (393 KB) for 2 cores Oct 04 14:13:01.127859 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: *** Operational MODE: preforking *** Oct 04 14:13:01.128068 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: *** uWSGI is running in multiple interpreter mode *** Oct 04 14:13:01.128068 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: spawned uWSGI master process (pid: 81447) Oct 04 14:13:01.128392 dv-cinder-860260-1-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 04 14:13:01.129647 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: spawned uWSGI worker 1 (pid: 81460, cores: 1) Oct 04 14:13:01.129647 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: spawned uWSGI worker 2 (pid: 81461, cores: 1) Oct 04 14:13:01.129647 dv-cinder-860260-1-iscsi devstack@keystone.service[81447]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 04 14:13:06.292770 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 04 14:13:06.292770 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: from cryptography.utils import int_from_bytes Oct 04 14:13:06.293588 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 04 14:13:06.293588 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: from cryptography.utils import int_from_bytes Oct 04 14:13:06.381089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 04 14:13:06.381410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 04 14:13:06.381690 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] command line args: [] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 04 14:13:06.382156 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 04 14:13:06.382452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ================================================================================ {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 04 14:13:06.382888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] admin_token = **** {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.383312 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] config_dir = [] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.383628 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.384012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] config_source = [] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.384366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] debug = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.384737 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.387002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.387002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.387002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.387002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.387387 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] list_limit = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.387685 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] log_config_append = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.387976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.390303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] log_dir = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.390303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] log_file = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.390303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.390303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.390303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.392383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.392812 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.393152 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.393639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.394075 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.395607 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.396055 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.396504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.396913 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.397354 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.397786 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.398308 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.399154 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.64/identity {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.399466 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] publish_errors = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.403965 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.404391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.404710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.405153 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.405564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.405976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] standard_threads = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.406380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.406808 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.410575 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.411269 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] use_journal = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.411571 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] use_json = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.411880 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] use_stderr = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.412185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] use_syslog = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.412652 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.413110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.413523 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.413978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.414392 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.414814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.415285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.415720 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.416096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.416529 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.416946 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.417382 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.417788 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.418225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.418640 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.419072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.419508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.419924 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.420403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.420848 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.421365 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.421782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.422233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.422687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.423127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.423565 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.423973 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.424401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.424799 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.425211 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.425636 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.426070 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.426469 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.426881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.427305 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.427740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.428158 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.428641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.429057 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.429447 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.429828 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.430207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.430593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.431522 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.431918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.432356 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.432792 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.433206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.433623 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.434020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.434437 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.434854 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.435280 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.435690 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.436107 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.436566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.436978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.437388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.437800 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.438193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.438619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.439027 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.439439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.439843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.440289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.440728 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.441129 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.441540 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.441945 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.442411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.442826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.443254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.443683 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.444092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.444535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.444949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.445376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.445791 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.446196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.447255 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.447678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.448108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.448557 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.448984 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.449385 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.449797 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.450210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.450732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.451186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.451532 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.451968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.452414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.452861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.454027 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.454027 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.454288 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.454445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.454844 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.455239 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.455654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.456019 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.456458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.456868 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.457293 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.457704 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.458138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.458604 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.459022 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.459423 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] auth.external = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.459822 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.460383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.460768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.461152 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] auth.password = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.461542 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] auth.token = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.461945 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.462385 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.462810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.463222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.463652 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.464085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.464524 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.464945 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] credential.caching = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.465381 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.465809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.466248 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.466692 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.467120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.467533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.467958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.468425 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.468838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.469272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.469765 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.470191 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.470645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.471067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.471509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.471918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] federation.caching = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.472359 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.472782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.473213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.473612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.474051 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.474464 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.474905 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.475314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.475752 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.476162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.476629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.477047 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] identity.caching = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.477482 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.477956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.478355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.478761 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.479194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.479596 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.480006 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.480488 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.480917 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.481317 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.481711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.482224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.482536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.482977 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.483427 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.483855 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.484287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.484794 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.485247 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.485654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.486055 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.486476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.486871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.487297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.487735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.488165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.488840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.489251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.489670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.490104 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.490518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.490954 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.491392 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.491805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.492230 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.492670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.493096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.494062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.494498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.494908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.495333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.495788 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.496228 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.496664 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.496921 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 04 14:13:06.496921 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: from cryptography.utils import int_from_bytes Oct 04 14:13:06.497514 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.497629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 04 14:13:06.497629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: from cryptography.utils import int_from_bytes Oct 04 14:13:06.497841 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.497964 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.498306 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.498725 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.499140 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.499546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.499968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.500459 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.500861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.501282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.501715 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.502136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.502557 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.502960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.503379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.503793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.504216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.504656 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.505097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.505520 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.505942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.506365 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.506790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.507195 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.507627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.508037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.508481 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.508940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.509970 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.510399 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.510838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.511254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.511689 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.512103 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.512572 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.512985 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.513394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.513805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.514241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.514659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.515072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.515487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.515918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] resource.caching = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.516371 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.516820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.517233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.517682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.518127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.518545 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.518980 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.519401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.519822 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.520237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] role.caching = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.520693 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] role.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.521109 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.521546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.521983 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.522425 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.522864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.523303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.523742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.524188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.524704 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.525111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.525550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.525988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.526428 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.526867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.527586 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.527996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.528462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.528898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.529331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.529775 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.530185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.530622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.531059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.531492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.531902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.532346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.532746 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.533191 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.533640 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.534074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.534495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.535178 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.535590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.535999 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] token.caching = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.536452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.536883 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.537304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.537756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.538187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.538615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.539048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.539477 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.539933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.540413 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.540837 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.541261 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.541693 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.542173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.542578 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.543061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.543480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.543940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.connection = **** {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.544389 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.544819 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.545236 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.545661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.546083 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.546506 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.546928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.547335 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.547757 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.548174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.548626 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.549041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.549479 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.549891 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.550310 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.550713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.551148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.551562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.551860 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81460) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 04 14:13:06.583526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 04 14:13:06.583795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 04 14:13:06.584090 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] command line args: [] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 04 14:13:06.584567 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 04 14:13:06.584851 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ================================================================================ {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 04 14:13:06.585264 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] admin_token = **** {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.585669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] config_dir = [] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.585969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.586265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] config_source = [] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.586563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] debug = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.586899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.587335 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.587626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.588048 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.588490 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.588870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] list_limit = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.589168 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] log_config_append = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.589474 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.589782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] log_dir = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.590089 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] log_file = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.590503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.590936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.591344 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.591760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.592055 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.592376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.592783 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.593191 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.593577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.593962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.594356 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.594745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.595152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.595560 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.596021 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.596886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.64/identity {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.597188 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] publish_errors = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.597570 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.597950 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.598247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.598648 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.599040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.599438 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] standard_threads = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.599828 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.600231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.600563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.600869 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] use_journal = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.601178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] use_json = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.601474 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] use_stderr = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.601767 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] use_syslog = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.602164 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 14:13:06.602660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.603040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.603494 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.603916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.604364 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.604774 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.605196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.605598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.606008 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.606433 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.606859 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.607259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.607668 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.608068 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.608521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.608943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.609341 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.609756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.609855 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:06.610156 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:06.610246 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.610511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:06.610611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.610956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.611382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.611830 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.612274 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.612744 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.613148 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.613532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.613918 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.614322 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.614755 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.615173 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.615564 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.615944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.616393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.616814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.617226 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.617657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.618057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.619098 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.619502 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.620078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.620518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.620930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.621324 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.621738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.622167 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.622554 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.622949 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.623339 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.623745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.624154 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.624611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.625005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.625402 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.625806 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.626222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.626634 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.627036 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.627424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.627537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:06.627850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.628255 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.628694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.629101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.629561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.629952 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.630345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.630746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.631146 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.631602 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.632005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.632452 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.632872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.633272 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.633670 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.635183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.635622 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.636030 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.636464 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.636881 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.637281 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.637710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.638119 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.638529 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.638932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.639300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.639668 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.640081 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.640515 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.640939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.641336 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.641742 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.642165 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.642575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.642987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.643399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.643800 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.644196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.644620 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.645030 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.645435 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.645850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.646235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.646641 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.647040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.647475 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.647914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.648356 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.648772 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] auth.external = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.649151 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.649704 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.650094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.650491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] auth.password = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.650907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] auth.token = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.651315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.651719 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.652134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.652561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.652990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.653403 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.653793 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.654189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] credential.caching = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.654620 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.655055 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.655476 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.655907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.656382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.656747 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.657161 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.657582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.657982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.658420 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.658889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.659307 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.660009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.660472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.660920 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.661325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] federation.caching = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.661736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.662158 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.662569 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.662957 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.663371 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.663774 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.664206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.664650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.665083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.665543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.665981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.666380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] identity.caching = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.666792 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.667217 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.667635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.668048 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.668509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.668896 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.669304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.669724 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.670141 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.670528 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.670916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.671315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.671731 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.672145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.672608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.673051 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.673473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.673932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.674338 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.674729 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.675120 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.675528 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.675914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.676372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.676752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.677162 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.677612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.678006 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.678429 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.678850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.679247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.679656 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.680070 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.680502 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.681474 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.683875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:06.684276 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:06.684525 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.686759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.687194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.687819 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.690097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.690619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.691084 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.691512 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.691917 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.692359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.692779 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.693208 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.693626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.694048 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.694672 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.695073 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.695489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.695927 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.696334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.697660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.698090 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.698510 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.698916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.699386 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.699807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.700228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.700691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.701085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.701511 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.701932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.702363 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.702799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.703235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.703636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.704072 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.704519 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.704926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.705332 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.705741 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.706163 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.706603 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.707015 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.707440 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.707849 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.708285 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.708733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.709149 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.709558 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.709993 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.710410 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.710810 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.711214 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.711631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] resource.caching = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.712537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.712996 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.713549 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.713943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.714380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.714782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.715211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.715631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.716038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.716475 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] role.caching = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.716891 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] role.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.717281 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.717700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.718131 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.718540 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.718960 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.719374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.719804 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.720233 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.720692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.721092 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.721506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.721916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.722335 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.722763 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.723464 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.723860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.724289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.724766 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.725189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.725611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.726006 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.726426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.726845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.727255 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.728589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.729000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.729407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.729844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.730278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.730707 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.731124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.731547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.731955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.732372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] token.caching = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.732776 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.733212 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.733622 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.734047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.734468 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.734882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.735310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.735738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.736187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.736660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.737077 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.737464 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.737883 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.738364 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.738758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.739167 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.739619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.740048 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.connection = **** {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.740503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.740926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.741330 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.741727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.742130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.742533 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.742966 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.743978 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.744477 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.744910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.745332 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.745753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.746199 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.746655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.747085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.747478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.747890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.748302 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 14:13:06.748630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81461) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 04 14:13:06.795482 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:06.795687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:06.795911 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:06.796246 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:06.812026 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:06.844533 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:06.844875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:06.886137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:06.886490 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:06.913076 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:06.977057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:06.977407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.005991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.006348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.006711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.007066 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.007412 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.007770 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.181476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.181838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.182110 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.182237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.182475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.182593 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.182821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.182940 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.183157 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.183274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.183496 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.183613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.183825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.204257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.206031 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.206392 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.206765 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.207126 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.207460 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.207787 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.208122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.222668 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.226804 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.227154 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.227513 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.228457 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.228614 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.228807 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.228926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.229150 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.229269 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.229491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81460) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.229607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.229817 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.230152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.230490 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81461) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 14:13:07.258646 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.258871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.259007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 04 14:13:07.259120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 04 14:13:07.259581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.259906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 04 14:13:07.260135 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.260370 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.260493 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 04 14:13:07.260891 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.261246 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.261384 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.261663 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.261782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.262067 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.262182 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.279781 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.280047 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.280224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.280380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.280668 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.280787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.281095 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.281211 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.294834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.295066 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.300057 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.300363 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.300545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.300674 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.300979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.301096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.301422 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.301552 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.301867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.301983 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.302274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.302391 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.302694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.302811 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.303134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.303355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.303587 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.303798 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.304047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.304256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.304531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.304730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.347191 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.347782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.349772 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.350230 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.350352 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.350792 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.358882 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.359333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.359452 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.359775 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.359893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.360281 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.371997 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.372435 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.372559 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.372901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.379992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.380408 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.380580 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.380745 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.380924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.381082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.381325 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.381485 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.381748 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.381898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.382163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.382569 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.385357 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.385556 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.385556 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.386009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.411565 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.412023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.414732 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.415680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.419156 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.423103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.425029 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.425516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.428793 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.429268 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.433786 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.437641 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.439320 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.443318 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.445070 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.445552 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.446009 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.447158 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.447610 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.448066 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.449755 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.450234 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.450676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.451299 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.451748 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.452259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.480105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.480617 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.481035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.481443 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.482610 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.483087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.483493 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.483913 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.493684 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.496446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.496598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.499106 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.499370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.499832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.502039 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.502479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.504125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.504612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.505235 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.505701 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.506153 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.507219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.507697 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.508129 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.508623 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.509743 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.527486 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.527957 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.528583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.529037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.529499 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.529954 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.530411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.531395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.531878 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.532324 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.532779 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.533242 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.533682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.534145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.560066 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.563563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.565333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.569114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.570476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.573569 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.575381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.576213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.576693 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.577133 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.577892 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.580609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.581189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.581663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.588631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.591173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.593155 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.595647 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.596367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.596829 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.597274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.597695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.600839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.601310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.601756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.602206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.612708 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.613249 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.613693 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.618922 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.619610 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.620058 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.626190 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81460) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.626656 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.627102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.627543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.627968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.628442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.628911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.629381 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.629832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.630302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.630772 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.631216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.631648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.632088 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81460) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.632668 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81461) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 14:13:07.633129 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.633566 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.634557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.634989 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.635431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.635872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.636304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.636801 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.637260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.637705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.638157 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.638597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.639029 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81461) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 14:13:07.695267 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x564023f029c0 pid: 81460 (default app) Oct 04 14:13:07.702045 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b1027d-4874-47e3-a974-729a6711254c None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:07.702510 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b1027d-4874-47e3-a974-729a6711254c None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:07.702951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b1027d-4874-47e3-a974-729a6711254c None None] PATH_INFO: `/v3` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:07.704792 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x564023f029c0 pid: 81461 (default app) Oct 04 14:13:07.708341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1/1] 10.222.0.64 () {50 vars in 791 bytes} [Tue Oct 4 14:13:07 2022] GET /identity/v3/ => generated 255 bytes in 12 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:15.125978 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6455c3-adac-4777-9cbf-e2b75c41d20d None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:15.126608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6455c3-adac-4777-9cbf-e2b75c41d20d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:15.126608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6455c3-adac-4777-9cbf-e2b75c41d20d None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:15.131001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1/2] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:15 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:13:15.140943 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a26b8e-9ba0-4319-ab4c-439028c9b25a None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:15.141572 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a26b8e-9ba0-4319-ab4c-439028c9b25a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:15.141922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a26b8e-9ba0-4319-ab4c-439028c9b25a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:15.226800 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: INFO dbcounter [None req-08a26b8e-9ba0-4319-ab4c-439028c9b25a None None] Registered counter for database keystone Oct 04 14:13:15.248777 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG oslo_db.sqlalchemy.engines [None req-08a26b8e-9ba0-4319-ab4c-439028c9b25a None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81460) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 04 14:13:15.255931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writer thread running {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 04 14:13:15.451679 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG passlib.handlers.bcrypt [None req-08a26b8e-9ba0-4319-ab4c-439028c9b25a None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81460) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 04 14:13:15.452237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG passlib.handlers.bcrypt [None req-08a26b8e-9ba0-4319-ab4c-439028c9b25a None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81460) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 04 14:13:15.513622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-08a26b8e-9ba0-4319-ab4c-439028c9b25a None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:15.515298 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-08a26b8e-9ba0-4319-ab4c-439028c9b25a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:15.673511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 2/3] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:15 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 535 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 04 14:13:16.196429 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6455c3-adac-4777-9cbf-e2b75c41d20d None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:16.270380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: INFO dbcounter [None req-4d6455c3-adac-4777-9cbf-e2b75c41d20d None None] Registered counter for database keystone Oct 04 14:13:16.295180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG oslo_db.sqlalchemy.engines [None req-4d6455c3-adac-4777-9cbf-e2b75c41d20d None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81461) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 04 14:13:16.302507 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writer thread running {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 04 14:13:16.435550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6455c3-adac-4777-9cbf-e2b75c41d20d None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:16.440778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf00678-5351-4705-a35f-e96b19500419 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:16.441489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf00678-5351-4705-a35f-e96b19500419 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:16.441817 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf00678-5351-4705-a35f-e96b19500419 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:16.442153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf00678-5351-4705-a35f-e96b19500419 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:16.455065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf00678-5351-4705-a35f-e96b19500419 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:16.509845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: WARNING keystone.common.rbac_enforcer.enforcer [None req-4bf00678-5351-4705-a35f-e96b19500419 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 04 14:13:16.510182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf00678-5351-4705-a35f-e96b19500419 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:16.521292 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 2/4] 10.222.0.64 () {58 vars in 1324 bytes} [Tue Oct 4 14:13:16 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 14:13:16.527864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a26b8e-9ba0-4319-ab4c-439028c9b25a None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:16.548630 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a26b8e-9ba0-4319-ab4c-439028c9b25a None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:16.553532 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dbfd67-a476-494e-a8ed-5a39d15705a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:16.554707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dbfd67-a476-494e-a8ed-5a39d15705a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:16.554991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dbfd67-a476-494e-a8ed-5a39d15705a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:16.555974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dbfd67-a476-494e-a8ed-5a39d15705a1 admin admin] PATH_INFO: `/v3/projects/79763f634e4d4d53967547cda84e5ebd` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:16.558323 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03dbfd67-a476-494e-a8ed-5a39d15705a1 admin admin] RBAC: Authorizing `identity:get_project(project_id=79763f634e4d4d53967547cda84e5ebd)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:16.615341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: WARNING keystone.common.rbac_enforcer.enforcer [None req-03dbfd67-a476-494e-a8ed-5a39d15705a1 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 04 14:13:16.615684 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03dbfd67-a476-494e-a8ed-5a39d15705a1 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:16.622166 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 3/5] 10.222.0.64 () {56 vars in 1207 bytes} [Tue Oct 4 14:13:16 2022] GET /identity/v3/projects/79763f634e4d4d53967547cda84e5ebd => generated 344 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:20.170121 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbafb25-27c6-45dc-8cb5-efeeb51d5107 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:20.170703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbafb25-27c6-45dc-8cb5-efeeb51d5107 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:20.170703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbafb25-27c6-45dc-8cb5-efeeb51d5107 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:20.174905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 3/6] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:20 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:20.181356 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9321ba-aa8b-471e-b86d-3b6411e558bd None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:20.181920 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9321ba-aa8b-471e-b86d-3b6411e558bd None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:20.182030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9321ba-aa8b-471e-b86d-3b6411e558bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:20.204785 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-8b9321ba-aa8b-471e-b86d-3b6411e558bd None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:20.206380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8b9321ba-aa8b-471e-b86d-3b6411e558bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:20.215187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 4/7] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:20 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 04 14:13:20.501358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbafb25-27c6-45dc-8cb5-efeeb51d5107 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:20.514642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbafb25-27c6-45dc-8cb5-efeeb51d5107 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:20.519000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf17521e-f8f4-4415-b42a-cad0d2dde33e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:20.519617 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf17521e-f8f4-4415-b42a-cad0d2dde33e admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:20.519896 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf17521e-f8f4-4415-b42a-cad0d2dde33e admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:20.520171 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf17521e-f8f4-4415-b42a-cad0d2dde33e admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:20.529609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf17521e-f8f4-4415-b42a-cad0d2dde33e admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:20.531415 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf17521e-f8f4-4415-b42a-cad0d2dde33e admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:20.542746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 4/8] 10.222.0.64 () {58 vars in 1324 bytes} [Tue Oct 4 14:13:20 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 14:13:20.548641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9321ba-aa8b-471e-b86d-3b6411e558bd None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:20.558570 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9321ba-aa8b-471e-b86d-3b6411e558bd None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:20.563532 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594302aa-1785-45a1-84c8-126d4a9a70b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:20.564192 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594302aa-1785-45a1-84c8-126d4a9a70b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:20.564502 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594302aa-1785-45a1-84c8-126d4a9a70b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:20.564778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594302aa-1785-45a1-84c8-126d4a9a70b4 admin admin] PATH_INFO: `/v3/users/4fff265355db498da98d8cb52b62f573` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:20.566198 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594302aa-1785-45a1-84c8-126d4a9a70b4 admin admin] RBAC: Authorizing `identity:get_user(user_id=4fff265355db498da98d8cb52b62f573)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:20.568084 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594302aa-1785-45a1-84c8-126d4a9a70b4 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:20.569931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 5/9] 10.222.0.64 () {56 vars in 1198 bytes} [Tue Oct 4 14:13:20 2022] GET /identity/v3/users/4fff265355db498da98d8cb52b62f573 => generated 247 bytes in 22 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:22.948803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6184cf-c01f-4652-9793-0aadfd93ac4f None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:22.949673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6184cf-c01f-4652-9793-0aadfd93ac4f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:22.949673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6184cf-c01f-4652-9793-0aadfd93ac4f None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:22.950694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 5/10] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:13:22.957356 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea625988-e0bd-4b7d-b656-46da9994998c None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:22.958163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea625988-e0bd-4b7d-b656-46da9994998c None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:22.958321 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea625988-e0bd-4b7d-b656-46da9994998c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:22.978528 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-ea625988-e0bd-4b7d-b656-46da9994998c None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:22.979830 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ea625988-e0bd-4b7d-b656-46da9994998c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:23.002288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19565cb-02f0-46ad-941c-3599d4a30a0c None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:23.002713 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19565cb-02f0-46ad-941c-3599d4a30a0c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:23.003185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19565cb-02f0-46ad-941c-3599d4a30a0c None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:23.005948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 6/11] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:23 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:13:23.013292 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93124e48-30f5-40c7-8f6d-856072894916 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:23.013706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93124e48-30f5-40c7-8f6d-856072894916 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:23.014175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93124e48-30f5-40c7-8f6d-856072894916 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:23.061696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG passlib.handlers.bcrypt [None req-93124e48-30f5-40c7-8f6d-856072894916 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81461) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 04 14:13:23.062084 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG passlib.handlers.bcrypt [None req-93124e48-30f5-40c7-8f6d-856072894916 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81461) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 04 14:13:23.085656 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 6/12] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:13:23.111608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-93124e48-30f5-40c7-8f6d-856072894916 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:23.113239 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-93124e48-30f5-40c7-8f6d-856072894916 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:23.230505 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 7/13] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:13:23.364469 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea625988-e0bd-4b7d-b656-46da9994998c None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:23.392592 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:23.491364 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea625988-e0bd-4b7d-b656-46da9994998c None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:23.509491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93124e48-30f5-40c7-8f6d-856072894916 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:23.537273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e89606-5cb8-4365-b010-fe17e229980e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:23.538062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e89606-5cb8-4365-b010-fe17e229980e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:23.538431 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e89606-5cb8-4365-b010-fe17e229980e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:23.538796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e89606-5cb8-4365-b010-fe17e229980e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:23.553712 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e89606-5cb8-4365-b010-fe17e229980e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:13:23.564197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-b6e89606-5cb8-4365-b010-fe17e229980e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:13:23.564197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:13:23.564197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:13:23.564197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:13:23.564197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:13:23.564197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:13:23.564197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:13:23.564197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:13:23.564197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:13:23.564197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:13:23.564197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:13:23.564197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:13:23.564197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:13:23.564197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:13:23.564197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:13:23.564197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:13:23.566159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:13:23.566159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:13:23.566159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:13:23.566159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:13:23.566159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:13:23.566159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:13:23.566159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:13:23.566159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:13:23.566159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:13:23.566159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:13:23.566159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:13:23.566159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:13:23.566159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:13:23.566159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:13:23.566159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:13:23.566159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:13:23.567766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:13:23.567766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:13:23.567766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:13:23.567766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:13:23.567766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:13:23.567766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:13:23.567766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:13:23.567766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:13:23.567766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:13:23.567766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:13:23.567766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:13:23.567766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:13:23.567766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 7/14] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:13:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:13:23.576085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e89606-5cb8-4365-b010-fe17e229980e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:23.656839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93124e48-30f5-40c7-8f6d-856072894916 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:23.661182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:23.675644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:23.696688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e89606-5cb8-4365-b010-fe17e229980e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:23.709918 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69bc37c-acb1-4f8c-aae6-ef9f55665dd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:23.711467 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69bc37c-acb1-4f8c-aae6-ef9f55665dd2 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:23.711839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69bc37c-acb1-4f8c-aae6-ef9f55665dd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:23.712202 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69bc37c-acb1-4f8c-aae6-ef9f55665dd2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:23.737588 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f208069d-b2b9-4912-8151-84a045fa88a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:23.738225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f208069d-b2b9-4912-8151-84a045fa88a6 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:23.738511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f208069d-b2b9-4912-8151-84a045fa88a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:23.738785 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f208069d-b2b9-4912-8151-84a045fa88a6 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:23.739510 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f208069d-b2b9-4912-8151-84a045fa88a6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:23.739830 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f208069d-b2b9-4912-8151-84a045fa88a6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:23.741353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f208069d-b2b9-4912-8151-84a045fa88a6 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:23.757220 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 8/15] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:13:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:13:23.766481 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f208069d-b2b9-4912-8151-84a045fa88a6 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:23.819027 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69bc37c-acb1-4f8c-aae6-ef9f55665dd2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:23.820709 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69bc37c-acb1-4f8c-aae6-ef9f55665dd2 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:23.925881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f208069d-b2b9-4912-8151-84a045fa88a6 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:23.945914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 8/16] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:13:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 14:13:23.951992 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69bc37c-acb1-4f8c-aae6-ef9f55665dd2 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:23.955358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:23.973543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26adae3c-b131-4e46-80ed-6ababbcc8b97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:23.974323 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26adae3c-b131-4e46-80ed-6ababbcc8b97 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:23.974716 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26adae3c-b131-4e46-80ed-6ababbcc8b97 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:23.975112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26adae3c-b131-4e46-80ed-6ababbcc8b97 None admin] PATH_INFO: `/v3/users/4fff265355db498da98d8cb52b62f573` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:23.977235 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26adae3c-b131-4e46-80ed-6ababbcc8b97 None admin] RBAC: Authorizing `identity:get_user(user_id=4fff265355db498da98d8cb52b62f573)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:23.979008 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26adae3c-b131-4e46-80ed-6ababbcc8b97 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:23.980951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 9/17] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:13:23 2022] GET /identity/v3/users/4fff265355db498da98d8cb52b62f573 => generated 247 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:13:23.987698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26adae3c-b131-4e46-80ed-6ababbcc8b97 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:24.005835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:24.082846 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69bc37c-acb1-4f8c-aae6-ef9f55665dd2 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:24.123048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26adae3c-b131-4e46-80ed-6ababbcc8b97 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:24.125509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966d57b7-6fad-44f8-9cfb-e9ac2626c446 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:24.126542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966d57b7-6fad-44f8-9cfb-e9ac2626c446 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:24.126905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966d57b7-6fad-44f8-9cfb-e9ac2626c446 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:24.127272 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966d57b7-6fad-44f8-9cfb-e9ac2626c446 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:24.149076 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966d57b7-6fad-44f8-9cfb-e9ac2626c446 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:24.150590 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966d57b7-6fad-44f8-9cfb-e9ac2626c446 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:24.161491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9d3ba1-59ed-4b1f-b8ed-91fe5916db75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:24.162139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9d3ba1-59ed-4b1f-b8ed-91fe5916db75 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:24.162420 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9d3ba1-59ed-4b1f-b8ed-91fe5916db75 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:24.162692 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9d3ba1-59ed-4b1f-b8ed-91fe5916db75 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:24.164527 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9d3ba1-59ed-4b1f-b8ed-91fe5916db75 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:24.166033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9d3ba1-59ed-4b1f-b8ed-91fe5916db75 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:24.167718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 10/18] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:13:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:24.173722 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9d3ba1-59ed-4b1f-b8ed-91fe5916db75 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:24.180890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-966d57b7-6fad-44f8-9cfb-e9ac2626c446 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:13:24.180890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:13:24.180890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:13:24.180890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:13:24.180890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:13:24.180890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:13:24.180890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:13:24.180890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:13:24.180890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:13:24.180890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:13:24.180890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:13:24.180890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:13:24.180890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:13:24.180890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:13:24.180890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:13:24.180890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:13:24.181939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:13:24.181939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:13:24.181939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:13:24.181939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:13:24.181939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:13:24.181939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:13:24.181939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:13:24.181939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:13:24.181939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:13:24.181939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:13:24.181939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:13:24.181939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:13:24.181939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:13:24.181939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:13:24.181939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:13:24.181939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:13:24.181939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:13:24.182897 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:13:24.182897 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:13:24.182897 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:13:24.182897 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 9/19] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:13:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:13:24.190197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966d57b7-6fad-44f8-9cfb-e9ac2626c446 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:24.269595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:24.281805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:24.305396 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9d3ba1-59ed-4b1f-b8ed-91fe5916db75 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:24.327688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966d57b7-6fad-44f8-9cfb-e9ac2626c446 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:24.355651 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1805f938-755d-44b9-a2fb-682a5916093e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:24.356200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1805f938-755d-44b9-a2fb-682a5916093e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:24.356514 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1805f938-755d-44b9-a2fb-682a5916093e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:24.356796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1805f938-755d-44b9-a2fb-682a5916093e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:24.357543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1805f938-755d-44b9-a2fb-682a5916093e None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.domain.id=default, user.id=4fff265355db498da98d8cb52b62f573) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:24.357897 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1805f938-755d-44b9-a2fb-682a5916093e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:24.359443 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1805f938-755d-44b9-a2fb-682a5916093e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:24.368094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 11/20] 10.222.0.64 () {56 vars in 1322 bytes} [Tue Oct 4 14:13:24 2022] GET /identity/v3/role_assignments?user.id=4fff265355db498da98d8cb52b62f573&scope.domain.id=default&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 233 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:13:24.378511 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b277dd-a177-4327-8c40-426f894a391d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:24.379596 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b277dd-a177-4327-8c40-426f894a391d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:24.380116 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b277dd-a177-4327-8c40-426f894a391d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:24.380644 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b277dd-a177-4327-8c40-426f894a391d None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:24.381914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b277dd-a177-4327-8c40-426f894a391d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:24.382463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b277dd-a177-4327-8c40-426f894a391d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:24.385992 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b277dd-a177-4327-8c40-426f894a391d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:24.423862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 10/21] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:13:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:13:27.387444 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b198e6-dae8-4d11-a79f-8d72c006734c None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:27.387983 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b198e6-dae8-4d11-a79f-8d72c006734c None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:27.388555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b198e6-dae8-4d11-a79f-8d72c006734c None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:27.390916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 12/22] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:27 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:27.400367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c397b3f6-d2b3-430c-bf36-ed1e5fe0301f None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:27.401122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c397b3f6-d2b3-430c-bf36-ed1e5fe0301f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:27.401418 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c397b3f6-d2b3-430c-bf36-ed1e5fe0301f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:27.455426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-c397b3f6-d2b3-430c-bf36-ed1e5fe0301f None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:27.457319 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c397b3f6-d2b3-430c-bf36-ed1e5fe0301f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:27.568815 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 11/23] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:13:27.843745 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4b5b54-6583-4850-be2e-11f508d3cc50 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:27.844784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4b5b54-6583-4850-be2e-11f508d3cc50 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:27.845383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4b5b54-6583-4850-be2e-11f508d3cc50 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:27.847569 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 13/24] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:27 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:27.856479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f589d7d5-eb1a-4522-8fa1-711444c5cf3d None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:27.856763 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f589d7d5-eb1a-4522-8fa1-711444c5cf3d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:27.857249 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f589d7d5-eb1a-4522-8fa1-711444c5cf3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:27.885492 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-f589d7d5-eb1a-4522-8fa1-711444c5cf3d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:27.887014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f589d7d5-eb1a-4522-8fa1-711444c5cf3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:27.950077 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:27.988710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 12/25] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:13:28.184190 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2721ad0e-2575-441d-82f5-7af147443725 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:28.184907 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2721ad0e-2575-441d-82f5-7af147443725 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:28.185392 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2721ad0e-2575-441d-82f5-7af147443725 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:28.187035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 14/26] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:28.192980 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434d0dce-0c9b-42ba-af5a-c91d8dcc2ac8 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:28.193267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434d0dce-0c9b-42ba-af5a-c91d8dcc2ac8 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:28.193536 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434d0dce-0c9b-42ba-af5a-c91d8dcc2ac8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:28.222461 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-434d0dce-0c9b-42ba-af5a-c91d8dcc2ac8 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:28.223720 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-434d0dce-0c9b-42ba-af5a-c91d8dcc2ac8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:28.319275 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 13/27] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:13:28.366178 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2721ad0e-2575-441d-82f5-7af147443725 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:28.525606 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2721ad0e-2575-441d-82f5-7af147443725 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:28.575484 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b720f839-a738-49c0-82ac-364fa74618ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:28.576147 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b720f839-a738-49c0-82ac-364fa74618ac None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:28.576461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b720f839-a738-49c0-82ac-364fa74618ac None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:28.576737 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b720f839-a738-49c0-82ac-364fa74618ac None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:28.593378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b720f839-a738-49c0-82ac-364fa74618ac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:13:28.598467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-b720f839-a738-49c0-82ac-364fa74618ac None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:13:28.598467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:13:28.598467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:13:28.598467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:13:28.598467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:13:28.598467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:13:28.598467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:13:28.598467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:13:28.598467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:13:28.598467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:13:28.598467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:13:28.598467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:13:28.598467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:13:28.598467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:13:28.598467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:13:28.598467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:13:28.599405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:13:28.599405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:13:28.599405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:13:28.599405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:13:28.599405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:13:28.599405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:13:28.599405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:13:28.599405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:13:28.599405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:13:28.599405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:13:28.599405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:13:28.599405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:13:28.599405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:13:28.599405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:13:28.599405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:13:28.599405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:13:28.600174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:13:28.600174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:13:28.600174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:13:28.600174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:13:28.600174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:13:28.600174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:13:28.600174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:13:28.600174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:13:28.600174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:13:28.600174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:13:28.600174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:13:28.600174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:13:28.600174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 15/28] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:13:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:13:28.608823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434d0dce-0c9b-42ba-af5a-c91d8dcc2ac8 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:28.646299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b720f839-a738-49c0-82ac-364fa74618ac None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:28.652391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:28.931468 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b720f839-a738-49c0-82ac-364fa74618ac None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:28.945609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434d0dce-0c9b-42ba-af5a-c91d8dcc2ac8 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:28.961910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:28.988632 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e9ae47-5e5c-47f7-8ccd-58ceaa1e01ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:28.989657 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e9ae47-5e5c-47f7-8ccd-58ceaa1e01ae None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:28.990171 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e9ae47-5e5c-47f7-8ccd-58ceaa1e01ae None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:28.991498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e9ae47-5e5c-47f7-8ccd-58ceaa1e01ae None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:28.993439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e9ae47-5e5c-47f7-8ccd-58ceaa1e01ae None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:28.995182 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e9ae47-5e5c-47f7-8ccd-58ceaa1e01ae None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:29.050082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838e9abd-2a81-49ac-98d8-b3f43bf03166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:29.050748 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838e9abd-2a81-49ac-98d8-b3f43bf03166 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:29.051030 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838e9abd-2a81-49ac-98d8-b3f43bf03166 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:29.051304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838e9abd-2a81-49ac-98d8-b3f43bf03166 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:29.052003 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838e9abd-2a81-49ac-98d8-b3f43bf03166 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:29.052323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838e9abd-2a81-49ac-98d8-b3f43bf03166 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:29.053786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838e9abd-2a81-49ac-98d8-b3f43bf03166 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:29.087799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 14/29] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:13:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 479 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:29.088720 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838e9abd-2a81-49ac-98d8-b3f43bf03166 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:29.144749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 16/30] 10.222.0.64 () {60 vars in 1131 bytes} [Tue Oct 4 14:13:28 2022] POST /identity/v3/roles => generated 225 bytes in 499 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:13:29.147063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e9ae47-5e5c-47f7-8ccd-58ceaa1e01ae None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:29.239236 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:29.297818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e9ae47-5e5c-47f7-8ccd-58ceaa1e01ae None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:29.330609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838e9abd-2a81-49ac-98d8-b3f43bf03166 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:29.351630 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cd4aca-9769-4fc8-80a1-117ccdd0b951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:29.352464 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cd4aca-9769-4fc8-80a1-117ccdd0b951 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:29.352811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cd4aca-9769-4fc8-80a1-117ccdd0b951 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:29.353187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cd4aca-9769-4fc8-80a1-117ccdd0b951 None admin] PATH_INFO: `/v3/users/4fff265355db498da98d8cb52b62f573` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:29.355347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cd4aca-9769-4fc8-80a1-117ccdd0b951 None admin] RBAC: Authorizing `identity:get_user(user_id=4fff265355db498da98d8cb52b62f573)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:29.357703 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cd4aca-9769-4fc8-80a1-117ccdd0b951 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:29.364346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 17/31] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:13:29 2022] GET /identity/v3/users/4fff265355db498da98d8cb52b62f573 => generated 247 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:29.364346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbb6eb7-2b30-4334-a3d3-5565d15baedc None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:29.364346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbb6eb7-2b30-4334-a3d3-5565d15baedc None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:29.364346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbb6eb7-2b30-4334-a3d3-5565d15baedc None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:29.364346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 18/32] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:29.370096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3502d829-cf69-4ec1-8a59-4e47ebe16c42 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:29.370367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3502d829-cf69-4ec1-8a59-4e47ebe16c42 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:29.370754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3502d829-cf69-4ec1-8a59-4e47ebe16c42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:29.394305 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-3502d829-cf69-4ec1-8a59-4e47ebe16c42 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:29.396592 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-3502d829-cf69-4ec1-8a59-4e47ebe16c42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:29.418807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d605b8ac-d29f-446f-b3a2-914227cbac8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:29.420515 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d605b8ac-d29f-446f-b3a2-914227cbac8d None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:29.421342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d605b8ac-d29f-446f-b3a2-914227cbac8d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:29.422204 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d605b8ac-d29f-446f-b3a2-914227cbac8d None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:29.423866 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d605b8ac-d29f-446f-b3a2-914227cbac8d None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:29.427124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d605b8ac-d29f-446f-b3a2-914227cbac8d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:29.513084 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 19/33] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:13:29.514653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3502d829-cf69-4ec1-8a59-4e47ebe16c42 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:29.643483 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:29.702666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 15/34] 10.222.0.64 () {60 vars in 1131 bytes} [Tue Oct 4 14:13:29 2022] POST /identity/v3/roles => generated 219 bytes in 614 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:13:29.729419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3502d829-cf69-4ec1-8a59-4e47ebe16c42 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:29.798160 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf12344d-57e2-4b08-8a70-aef1c8a51d73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:29.799233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf12344d-57e2-4b08-8a70-aef1c8a51d73 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:29.799685 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf12344d-57e2-4b08-8a70-aef1c8a51d73 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:29.800179 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf12344d-57e2-4b08-8a70-aef1c8a51d73 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:29.802304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf12344d-57e2-4b08-8a70-aef1c8a51d73 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:29.805004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf12344d-57e2-4b08-8a70-aef1c8a51d73 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:29.807855 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 20/35] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:13:29 2022] GET /identity/v3/domains/default => generated 199 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:29.816912 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d605b8ac-d29f-446f-b3a2-914227cbac8d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:29.872747 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:29.985904 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf12344d-57e2-4b08-8a70-aef1c8a51d73 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:30.026952 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d605b8ac-d29f-446f-b3a2-914227cbac8d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:30.120675 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887c136d-0734-4dee-9817-e72c0bc0336e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:30.121439 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887c136d-0734-4dee-9817-e72c0bc0336e None admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:30.121728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887c136d-0734-4dee-9817-e72c0bc0336e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:30.122010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887c136d-0734-4dee-9817-e72c0bc0336e None admin] PATH_INFO: `/v3/domains/default/users/4fff265355db498da98d8cb52b62f573/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:30.124888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887c136d-0734-4dee-9817-e72c0bc0336e None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=4fff265355db498da98d8cb52b62f573, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:30.135156 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887c136d-0734-4dee-9817-e72c0bc0336e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:30.137552 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf12344d-57e2-4b08-8a70-aef1c8a51d73 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:30.167044 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 16/36] 10.222.0.64 () {58 vars in 1361 bytes} [Tue Oct 4 14:13:29 2022] PUT /identity/v3/domains/default/users/4fff265355db498da98d8cb52b62f573/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:13:30.188933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b900a6-c7cf-43ea-bb33-9fb98df2b47a None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:30.189377 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b900a6-c7cf-43ea-bb33-9fb98df2b47a None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:30.189799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b900a6-c7cf-43ea-bb33-9fb98df2b47a None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:30.191826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 17/37] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:30.202018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d6afb4-7ec5-43da-bf4c-f087feb37459 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:30.202520 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d6afb4-7ec5-43da-bf4c-f087feb37459 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:30.202941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d6afb4-7ec5-43da-bf4c-f087feb37459 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:30.208492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:30.225479 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f476c18-9b5d-4da2-8d4c-bb0c80eeec0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:30.225479 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f476c18-9b5d-4da2-8d4c-bb0c80eeec0b None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:30.225479 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f476c18-9b5d-4da2-8d4c-bb0c80eeec0b None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:30.225960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f476c18-9b5d-4da2-8d4c-bb0c80eeec0b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:30.232593 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-a4d6afb4-7ec5-43da-bf4c-f087feb37459 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:30.234520 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a4d6afb4-7ec5-43da-bf4c-f087feb37459 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:30.245655 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f476c18-9b5d-4da2-8d4c-bb0c80eeec0b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:30.249081 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f476c18-9b5d-4da2-8d4c-bb0c80eeec0b None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:30.281735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-3f476c18-9b5d-4da2-8d4c-bb0c80eeec0b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:13:30.281735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:13:30.281735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:13:30.281735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:13:30.281735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:13:30.281735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:13:30.281735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:13:30.281735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:13:30.281735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:13:30.281735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:13:30.281735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:13:30.281735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:13:30.281735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:13:30.281735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:13:30.281735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:13:30.281735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:13:30.282875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:13:30.282875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:13:30.282875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:13:30.282875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:13:30.282875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:13:30.282875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:13:30.282875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:13:30.282875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:13:30.282875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:13:30.282875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:13:30.282875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:13:30.282875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:13:30.282875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:13:30.282875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:13:30.282875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:13:30.282875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:13:30.282875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:13:30.284636 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:13:30.284636 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:13:30.284636 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:13:30.284636 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 21/38] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:13:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:13:30.286315 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8714039b-ecba-488f-b10c-76d0d9f7fbf0 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:30.286730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8714039b-ecba-488f-b10c-76d0d9f7fbf0 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:30.287139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8714039b-ecba-488f-b10c-76d0d9f7fbf0 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:30.289495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 22/39] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:30 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:30.297312 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8714039b-ecba-488f-b10c-76d0d9f7fbf0 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:30.363592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 18/40] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:13:30.365200 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c018a1-6b4a-4871-899e-7c7f311c555d None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:30.365665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c018a1-6b4a-4871-899e-7c7f311c555d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:30.366120 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c018a1-6b4a-4871-899e-7c7f311c555d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:30.404069 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-97c018a1-6b4a-4871-899e-7c7f311c555d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:30.405902 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-97c018a1-6b4a-4871-899e-7c7f311c555d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:30.436964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:30.471309 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8714039b-ecba-488f-b10c-76d0d9f7fbf0 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:30.528741 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 19/41] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:13:30.541644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed45109-730d-4bcf-b34f-2bb8675f6e16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:30.543123 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed45109-730d-4bcf-b34f-2bb8675f6e16 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:30.543961 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed45109-730d-4bcf-b34f-2bb8675f6e16 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:30.544747 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed45109-730d-4bcf-b34f-2bb8675f6e16 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:30.546312 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed45109-730d-4bcf-b34f-2bb8675f6e16 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:30.547126 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed45109-730d-4bcf-b34f-2bb8675f6e16 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:30.551595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed45109-730d-4bcf-b34f-2bb8675f6e16 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:30.588091 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 23/42] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:13:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:30.594022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c018a1-6b4a-4871-899e-7c7f311c555d None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:30.711851 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed45109-730d-4bcf-b34f-2bb8675f6e16 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:30.718336 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c018a1-6b4a-4871-899e-7c7f311c555d None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:30.758935 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971108f1-17af-4e4b-bae8-7ea6edc3a81d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:30.759577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971108f1-17af-4e4b-bae8-7ea6edc3a81d None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:30.759858 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971108f1-17af-4e4b-bae8-7ea6edc3a81d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:30.760133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971108f1-17af-4e4b-bae8-7ea6edc3a81d None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:30.760872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971108f1-17af-4e4b-bae8-7ea6edc3a81d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:30.762400 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971108f1-17af-4e4b-bae8-7ea6edc3a81d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:30.791203 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:30.802347 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 20/43] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Oct 4 14:13:30 2022] POST /identity/v3/projects => generated 301 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:13:30.813412 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:30.841567 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971108f1-17af-4e4b-bae8-7ea6edc3a81d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:30.949096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed45109-730d-4bcf-b34f-2bb8675f6e16 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:30.982561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971108f1-17af-4e4b-bae8-7ea6edc3a81d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:31.029375 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34700c5d-7e85-4608-9707-4b293c29808e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:31.030023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34700c5d-7e85-4608-9707-4b293c29808e None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:31.030315 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34700c5d-7e85-4608-9707-4b293c29808e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:31.030596 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34700c5d-7e85-4608-9707-4b293c29808e None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:31.031327 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34700c5d-7e85-4608-9707-4b293c29808e None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:31.035035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34700c5d-7e85-4608-9707-4b293c29808e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:31.039635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab304315-c4f1-4af3-ab95-6519b89326e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:31.039635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab304315-c4f1-4af3-ab95-6519b89326e7 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:31.039635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab304315-c4f1-4af3-ab95-6519b89326e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:31.039635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab304315-c4f1-4af3-ab95-6519b89326e7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:31.040219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab304315-c4f1-4af3-ab95-6519b89326e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:31.041500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab304315-c4f1-4af3-ab95-6519b89326e7 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:31.043246 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 21/44] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:13:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:31.050569 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab304315-c4f1-4af3-ab95-6519b89326e7 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:31.059092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 24/45] 10.222.0.64 () {60 vars in 1131 bytes} [Tue Oct 4 14:13:30 2022] POST /identity/v3/roles => generated 223 bytes in 348 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:13:31.155543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:31.199797 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab304315-c4f1-4af3-ab95-6519b89326e7 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:31.242236 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7aaa35-c95f-496c-9dba-8025e1b3e600 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:31.243081 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7aaa35-c95f-496c-9dba-8025e1b3e600 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:31.243408 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7aaa35-c95f-496c-9dba-8025e1b3e600 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:31.243687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7aaa35-c95f-496c-9dba-8025e1b3e600 None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:31.244415 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7aaa35-c95f-496c-9dba-8025e1b3e600 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:31.245964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7aaa35-c95f-496c-9dba-8025e1b3e600 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:31.277715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 22/46] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Oct 4 14:13:31 2022] POST /identity/v3/projects => generated 312 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:13:32.870487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25236459-3f39-4d10-b7e8-775f21ca3528 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:32.871052 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25236459-3f39-4d10-b7e8-775f21ca3528 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:32.871498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25236459-3f39-4d10-b7e8-775f21ca3528 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:32.874096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 25/47] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:32 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:32.884231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33884e36-3567-4ea0-acfd-097aee6f78be None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:32.884867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33884e36-3567-4ea0-acfd-097aee6f78be None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:32.885259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33884e36-3567-4ea0-acfd-097aee6f78be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:32.917482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-33884e36-3567-4ea0-acfd-097aee6f78be None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:32.918898 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-33884e36-3567-4ea0-acfd-097aee6f78be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:33.001110 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 23/48] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:13:33.470307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25236459-3f39-4d10-b7e8-775f21ca3528 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:33.653339 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25236459-3f39-4d10-b7e8-775f21ca3528 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:33.655405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:33.701158 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97b0b24-0a6c-4e9c-897d-6cf1f507e173 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:33.701995 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97b0b24-0a6c-4e9c-897d-6cf1f507e173 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:33.702354 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97b0b24-0a6c-4e9c-897d-6cf1f507e173 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:33.702703 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97b0b24-0a6c-4e9c-897d-6cf1f507e173 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:33.716097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97b0b24-0a6c-4e9c-897d-6cf1f507e173 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:13:33.721343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-e97b0b24-0a6c-4e9c-897d-6cf1f507e173 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:13:33.721343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:13:33.721343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:13:33.721343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:13:33.721343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:13:33.721343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:13:33.721343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:13:33.721343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:13:33.721343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:13:33.721343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:13:33.721343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:13:33.721343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:13:33.721343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:13:33.721343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:13:33.721343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:13:33.721343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:13:33.723442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:13:33.723442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:13:33.723442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:13:33.723442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:13:33.723442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:13:33.723442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:13:33.723442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:13:33.723442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:13:33.723442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:13:33.723442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:13:33.723442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:13:33.723442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:13:33.723442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:13:33.723442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:13:33.723442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:13:33.723442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:13:33.725232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:13:33.725232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:13:33.725232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:13:33.725232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:13:33.725232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:13:33.725232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:13:33.725232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:13:33.725232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:13:33.725232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:13:33.725232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:13:33.725232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:13:33.725232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:13:33.725232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 26/49] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:13:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:13:33.731077 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33884e36-3567-4ea0-acfd-097aee6f78be None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:33.736759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701d4bb9-6b6a-446b-8fb2-6ffcb7bb4eb7 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:33.737049 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701d4bb9-6b6a-446b-8fb2-6ffcb7bb4eb7 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:33.737856 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701d4bb9-6b6a-446b-8fb2-6ffcb7bb4eb7 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:33.739237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 27/50] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:33.745890 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708384ab-a8d5-4479-951b-5ec603670e8a None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:33.746310 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708384ab-a8d5-4479-951b-5ec603670e8a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:33.746744 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708384ab-a8d5-4479-951b-5ec603670e8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:33.767537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-708384ab-a8d5-4479-951b-5ec603670e8a None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:33.768835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-708384ab-a8d5-4479-951b-5ec603670e8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:33.805779 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:33.857052 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 28/51] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:13:33.892087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33884e36-3567-4ea0-acfd-097aee6f78be None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:33.953454 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666cb938-6a0b-429c-928e-14f0f107186b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:33.954239 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666cb938-6a0b-429c-928e-14f0f107186b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:33.954620 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666cb938-6a0b-429c-928e-14f0f107186b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:33.954994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666cb938-6a0b-429c-928e-14f0f107186b None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:33.955858 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666cb938-6a0b-429c-928e-14f0f107186b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:33.956276 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666cb938-6a0b-429c-928e-14f0f107186b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:33.957939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666cb938-6a0b-429c-928e-14f0f107186b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:33.969190 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 24/52] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:13:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:33.975321 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708384ab-a8d5-4479-951b-5ec603670e8a None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:34.136387 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666cb938-6a0b-429c-928e-14f0f107186b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:34.276618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:34.294469 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708384ab-a8d5-4479-951b-5ec603670e8a None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:34.356818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b753f4-5045-4430-bafc-47cc006bb93d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:34.357872 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b753f4-5045-4430-bafc-47cc006bb93d None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:34.358346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b753f4-5045-4430-bafc-47cc006bb93d None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:34.358820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b753f4-5045-4430-bafc-47cc006bb93d None admin] PATH_INFO: `/v3/users/4fff265355db498da98d8cb52b62f573` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:34.360975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b753f4-5045-4430-bafc-47cc006bb93d None admin] RBAC: Authorizing `identity:get_user(user_id=4fff265355db498da98d8cb52b62f573)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:34.363866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b753f4-5045-4430-bafc-47cc006bb93d None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:34.367499 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 29/53] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:13:33 2022] GET /identity/v3/users/4fff265355db498da98d8cb52b62f573 => generated 247 bytes in 393 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:34.376380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b753f4-5045-4430-bafc-47cc006bb93d None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:34.416624 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666cb938-6a0b-429c-928e-14f0f107186b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:34.439805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:34.473663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4318f572-d011-469d-9744-48b8bedd8364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:34.474316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4318f572-d011-469d-9744-48b8bedd8364 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:34.474591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4318f572-d011-469d-9744-48b8bedd8364 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:34.474870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4318f572-d011-469d-9744-48b8bedd8364 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:34.476760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4318f572-d011-469d-9744-48b8bedd8364 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:34.478265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4318f572-d011-469d-9744-48b8bedd8364 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:34.480012 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 25/54] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:13:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 345 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:13:34.486076 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4318f572-d011-469d-9744-48b8bedd8364 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:34.511724 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b753f4-5045-4430-bafc-47cc006bb93d None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:34.563390 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa7219b-2a6e-4df6-8bc3-9565558f43e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:34.564087 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa7219b-2a6e-4df6-8bc3-9565558f43e3 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:34.564509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa7219b-2a6e-4df6-8bc3-9565558f43e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:34.564886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa7219b-2a6e-4df6-8bc3-9565558f43e3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:34.566513 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa7219b-2a6e-4df6-8bc3-9565558f43e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:34.568301 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa7219b-2a6e-4df6-8bc3-9565558f43e3 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:34.571196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 30/55] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:13:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:34.578293 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa7219b-2a6e-4df6-8bc3-9565558f43e3 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:34.628976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:34.634308 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4318f572-d011-469d-9744-48b8bedd8364 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:34.672334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a7ff1a-9f3f-4287-924a-2a581e855be7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:34.672984 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a7ff1a-9f3f-4287-924a-2a581e855be7 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:34.673266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a7ff1a-9f3f-4287-924a-2a581e855be7 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:34.673541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a7ff1a-9f3f-4287-924a-2a581e855be7 None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:34.674218 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a7ff1a-9f3f-4287-924a-2a581e855be7 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:34.675726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a7ff1a-9f3f-4287-924a-2a581e855be7 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:34.705426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 26/56] 10.222.0.64 () {60 vars in 1140 bytes} [Tue Oct 4 14:13:34 2022] POST /identity/v3/projects => generated 298 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:13:34.765121 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa7219b-2a6e-4df6-8bc3-9565558f43e3 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:34.828407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edf8f32-2ef4-4d04-aa92-0fe8cde8dcfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:34.829264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edf8f32-2ef4-4d04-aa92-0fe8cde8dcfe None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:34.829755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edf8f32-2ef4-4d04-aa92-0fe8cde8dcfe None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:34.830185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edf8f32-2ef4-4d04-aa92-0fe8cde8dcfe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:34.831383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edf8f32-2ef4-4d04-aa92-0fe8cde8dcfe None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.domain.id=default, user.id=4fff265355db498da98d8cb52b62f573) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:34.832008 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edf8f32-2ef4-4d04-aa92-0fe8cde8dcfe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:34.834377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edf8f32-2ef4-4d04-aa92-0fe8cde8dcfe None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:34.845641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 31/57] 10.222.0.64 () {56 vars in 1322 bytes} [Tue Oct 4 14:13:34 2022] GET /identity/v3/role_assignments?user.id=4fff265355db498da98d8cb52b62f573&scope.domain.id=default&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 530 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:37.107341 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d3004b-2953-4765-8821-6e490c300dd5 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:37.107873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d3004b-2953-4765-8821-6e490c300dd5 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:37.107963 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d3004b-2953-4765-8821-6e490c300dd5 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:37.109387 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 27/58] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:37.115633 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10260913-f326-47d6-9c18-ea32b8c2cbd6 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:37.116779 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10260913-f326-47d6-9c18-ea32b8c2cbd6 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:37.116870 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10260913-f326-47d6-9c18-ea32b8c2cbd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:37.135698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-10260913-f326-47d6-9c18-ea32b8c2cbd6 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:37.136969 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-10260913-f326-47d6-9c18-ea32b8c2cbd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:37.211866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 32/59] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:13:37.490613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d3004b-2953-4765-8821-6e490c300dd5 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:37.528360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:37.627178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d3004b-2953-4765-8821-6e490c300dd5 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:37.668508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dbafa3-879c-4814-b385-aea72bce10ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:37.669164 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dbafa3-879c-4814-b385-aea72bce10ee None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:37.669448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dbafa3-879c-4814-b385-aea72bce10ee None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:37.669724 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dbafa3-879c-4814-b385-aea72bce10ee None admin] PATH_INFO: `/v3/domains/default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:37.670949 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dbafa3-879c-4814-b385-aea72bce10ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:37.672469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dbafa3-879c-4814-b385-aea72bce10ee None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:37.674146 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 28/60] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:13:37 2022] GET /identity/v3/domains/default => generated 199 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:37.681265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10260913-f326-47d6-9c18-ea32b8c2cbd6 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:37.709544 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:37.798990 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10260913-f326-47d6-9c18-ea32b8c2cbd6 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:37.839339 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bb7db2-4e0e-45cf-a4e2-07cbabd4dddb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:37.839995 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bb7db2-4e0e-45cf-a4e2-07cbabd4dddb None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:37.840378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bb7db2-4e0e-45cf-a4e2-07cbabd4dddb None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:37.840664 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bb7db2-4e0e-45cf-a4e2-07cbabd4dddb None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:37.841342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bb7db2-4e0e-45cf-a4e2-07cbabd4dddb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:37.842889 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bb7db2-4e0e-45cf-a4e2-07cbabd4dddb None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:37.875742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 33/61] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:13:37 2022] POST /identity/v3/users => generated 275 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:13:42.188162 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b674c1-aa12-45cd-b602-736ddf014849 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:42.188713 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b674c1-aa12-45cd-b602-736ddf014849 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:42.189159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b674c1-aa12-45cd-b602-736ddf014849 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:42.191730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 29/62] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:42 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:42.203427 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4f526d-61b8-4f02-a675-daa70ffed3e5 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:42.203977 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4f526d-61b8-4f02-a675-daa70ffed3e5 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:42.204445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4f526d-61b8-4f02-a675-daa70ffed3e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:42.250566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-9d4f526d-61b8-4f02-a675-daa70ffed3e5 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:42.252759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9d4f526d-61b8-4f02-a675-daa70ffed3e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:42.432611 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 34/63] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:13:42.836964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cee31b-56fc-4077-a69c-c24e7b2bb941 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:42.837977 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cee31b-56fc-4077-a69c-c24e7b2bb941 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:42.838721 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cee31b-56fc-4077-a69c-c24e7b2bb941 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:42.841676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 30/64] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:42 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:42.848001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2b7dbd-f09d-4b5e-a8ed-54e0de71b2a4 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:42.848285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2b7dbd-f09d-4b5e-a8ed-54e0de71b2a4 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:42.848597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2b7dbd-f09d-4b5e-a8ed-54e0de71b2a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:42.880012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-ad2b7dbd-f09d-4b5e-a8ed-54e0de71b2a4 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:42.881267 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ad2b7dbd-f09d-4b5e-a8ed-54e0de71b2a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:42.884101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:42.988660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b43e2fe-12f3-491d-a916-b2711439875b None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:42.989625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b43e2fe-12f3-491d-a916-b2711439875b None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:42.990379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b43e2fe-12f3-491d-a916-b2711439875b None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:42.993866 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 31/65] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:42 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:43.002943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8da000e-3df6-4b69-92ff-91ac90e9bf94 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:43.003770 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8da000e-3df6-4b69-92ff-91ac90e9bf94 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:43.004478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8da000e-3df6-4b69-92ff-91ac90e9bf94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:43.008058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 35/66] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:13:43.045160 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2b7dbd-f09d-4b5e-a8ed-54e0de71b2a4 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:43.076360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-a8da000e-3df6-4b69-92ff-91ac90e9bf94 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:43.081069 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a8da000e-3df6-4b69-92ff-91ac90e9bf94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:43.270942 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 32/67] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:13:43.307485 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8da000e-3df6-4b69-92ff-91ac90e9bf94 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:43.376576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2b7dbd-f09d-4b5e-a8ed-54e0de71b2a4 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:43.408107 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:43.444548 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8797bd1-2a02-4dff-b368-2f65d2933043 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:43.446057 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8797bd1-2a02-4dff-b368-2f65d2933043 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:43.446560 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8797bd1-2a02-4dff-b368-2f65d2933043 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:43.447076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8797bd1-2a02-4dff-b368-2f65d2933043 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:43.460400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:43.466555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8797bd1-2a02-4dff-b368-2f65d2933043 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:13:43.477698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-b8797bd1-2a02-4dff-b368-2f65d2933043 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:13:43.477698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:13:43.477698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:13:43.477698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:13:43.477698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:13:43.477698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:13:43.477698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:13:43.477698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:13:43.477698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:13:43.477698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:13:43.477698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:13:43.477698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:13:43.477698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:13:43.477698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:13:43.477698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:13:43.477698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:13:43.479408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:13:43.479408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:13:43.479408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:13:43.479408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:13:43.479408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:13:43.479408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:13:43.479408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:13:43.479408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:13:43.479408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:13:43.479408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:13:43.479408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:13:43.479408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:13:43.479408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:13:43.479408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:13:43.479408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:13:43.479408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:13:43.481015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:13:43.481015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:13:43.481015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:13:43.481015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:13:43.481015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:13:43.481015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:13:43.481015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:13:43.481015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:13:43.481015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:13:43.481015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:13:43.481015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:13:43.481015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:13:43.492477 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 36/68] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:13:43 2022] GET /identity/v3/roles/member => generated 84 bytes in 443 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:13:43.496209 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5c6f5f-2b5d-4e2e-8479-363d17675ac3 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:43.497143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5c6f5f-2b5d-4e2e-8479-363d17675ac3 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:43.497908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5c6f5f-2b5d-4e2e-8479-363d17675ac3 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:43.504837 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 37/69] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:43 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:43.506870 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5c6f5f-2b5d-4e2e-8479-363d17675ac3 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:43.514497 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8da000e-3df6-4b69-92ff-91ac90e9bf94 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:43.564575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578c95e4-9658-4977-aeab-a1b493037c0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:43.565477 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578c95e4-9658-4977-aeab-a1b493037c0b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:43.565964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578c95e4-9658-4977-aeab-a1b493037c0b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:43.566389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578c95e4-9658-4977-aeab-a1b493037c0b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:43.568115 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578c95e4-9658-4977-aeab-a1b493037c0b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:43.571710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578c95e4-9658-4977-aeab-a1b493037c0b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:43.573829 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 33/70] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:13:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:13:43.579184 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50376fc-88c2-44f4-a4a8-756210fc614e None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:43.579184 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50376fc-88c2-44f4-a4a8-756210fc614e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:43.579423 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50376fc-88c2-44f4-a4a8-756210fc614e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:43.607389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-b50376fc-88c2-44f4-a4a8-756210fc614e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:43.608942 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b50376fc-88c2-44f4-a4a8-756210fc614e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:43.681363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5c6f5f-2b5d-4e2e-8479-363d17675ac3 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:43.704366 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 34/71] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:13:43.706363 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50376fc-88c2-44f4-a4a8-756210fc614e None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:43.735671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a796567c-6acc-4195-b1e8-6e60470c12ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:43.736551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a796567c-6acc-4195-b1e8-6e60470c12ba None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:43.737051 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a796567c-6acc-4195-b1e8-6e60470c12ba None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:43.737510 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a796567c-6acc-4195-b1e8-6e60470c12ba None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:43.738571 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a796567c-6acc-4195-b1e8-6e60470c12ba None admin] RBAC: Adding query filter params (name=member) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:43.739073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a796567c-6acc-4195-b1e8-6e60470c12ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:43.741799 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a796567c-6acc-4195-b1e8-6e60470c12ba None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:43.753073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 38/72] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:13:43 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:43.754199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a796567c-6acc-4195-b1e8-6e60470c12ba None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:43.776792 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:43.854915 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:43.923117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50376fc-88c2-44f4-a4a8-756210fc614e None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:43.926275 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a796567c-6acc-4195-b1e8-6e60470c12ba None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:43.971505 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a91f07b-a6af-42b1-a13c-0e6a5bba9db7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:43.972393 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a91f07b-a6af-42b1-a13c-0e6a5bba9db7 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:43.972874 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a91f07b-a6af-42b1-a13c-0e6a5bba9db7 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:43.973353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a91f07b-a6af-42b1-a13c-0e6a5bba9db7 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:43.989992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a91f07b-a6af-42b1-a13c-0e6a5bba9db7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:13:43.994594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b82b96-9203-4d73-9889-e4c1268296f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:43.995582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b82b96-9203-4d73-9889-e4c1268296f2 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:43.996040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b82b96-9203-4d73-9889-e4c1268296f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:43.996522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b82b96-9203-4d73-9889-e4c1268296f2 None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:43.996968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-1a91f07b-a6af-42b1-a13c-0e6a5bba9db7 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:13:43.996968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:13:43.996968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:13:43.996968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:13:43.996968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:13:43.996968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:13:43.996968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:13:43.996968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:13:43.996968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:13:43.996968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:13:43.996968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:13:43.996968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:13:43.996968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:13:43.996968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:13:43.996968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:13:43.996968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:13:43.997933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:13:43.997933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:13:43.997933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:13:43.997933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:13:43.997933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:13:43.997933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:13:43.997933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:13:43.997933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:13:43.997933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:13:43.997933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:13:43.997933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:13:43.997933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:13:43.997933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:13:43.997933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:13:43.997933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:13:43.997933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:13:43.998718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:13:43.998718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:13:43.998718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:13:43.998718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:13:43.998718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:13:43.998718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:13:43.998718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:13:43.998718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:13:43.998718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:13:43.998718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:13:43.998718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:13:43.998718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:13:43.999300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b82b96-9203-4d73-9889-e4c1268296f2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:43.999366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 39/73] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:13:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:13:43.999669 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b52f57d-7fd5-46f4-8038-d35e5649d278 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:44.000230 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b52f57d-7fd5-46f4-8038-d35e5649d278 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:44.000694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b52f57d-7fd5-46f4-8038-d35e5649d278 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:44.001468 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b82b96-9203-4d73-9889-e4c1268296f2 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:44.004261 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 40/74] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:44.004261 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b52f57d-7fd5-46f4-8038-d35e5649d278 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:44.040046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 35/75] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:13:43 2022] POST /identity/v3/users => generated 289 bytes in 335 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:13:44.041304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b82b96-9203-4d73-9889-e4c1268296f2 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:44.195978 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:44.207568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b52f57d-7fd5-46f4-8038-d35e5649d278 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:44.225804 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b82b96-9203-4d73-9889-e4c1268296f2 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:44.252186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:44.280780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f711f5-2334-455a-af31-9d9639bc997c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:44.281924 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f711f5-2334-455a-af31-9d9639bc997c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:44.282433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f711f5-2334-455a-af31-9d9639bc997c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:44.282919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f711f5-2334-455a-af31-9d9639bc997c None admin] PATH_INFO: `/v3/users/ac210aede44f454dbb9c3aa4a942ffab` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:44.284959 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331040fc-d012-4656-922f-007df36f408f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:44.286026 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331040fc-d012-4656-922f-007df36f408f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:44.286481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331040fc-d012-4656-922f-007df36f408f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:44.286903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331040fc-d012-4656-922f-007df36f408f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:44.309565 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331040fc-d012-4656-922f-007df36f408f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:13:44.310487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f711f5-2334-455a-af31-9d9639bc997c None admin] RBAC: Authorizing `identity:get_user(user_id=ac210aede44f454dbb9c3aa4a942ffab)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:44.313641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f711f5-2334-455a-af31-9d9639bc997c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:44.323383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 41/76] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:13:44 2022] GET /identity/v3/users/ac210aede44f454dbb9c3aa4a942ffab => generated 275 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:44.323383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f711f5-2334-455a-af31-9d9639bc997c None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:44.323626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-331040fc-d012-4656-922f-007df36f408f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:13:44.323626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:13:44.323626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:13:44.323626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:13:44.323626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:13:44.323626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:13:44.323626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:13:44.323626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:13:44.323626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:13:44.323626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:13:44.323626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:13:44.323626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:13:44.323626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:13:44.323626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:13:44.323626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:13:44.323626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:13:44.324544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:13:44.324544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:13:44.324544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:13:44.324544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:13:44.324544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:13:44.324544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:13:44.324544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:13:44.324544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:13:44.324544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:13:44.324544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:13:44.324544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:13:44.324544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:13:44.324544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:13:44.324544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:13:44.324544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:13:44.324544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:13:44.325325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:13:44.325325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:13:44.325325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:13:44.325325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:13:44.325325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:13:44.325325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:13:44.325325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:13:44.325325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:13:44.325325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:13:44.325325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:13:44.325325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:13:44.325325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:13:44.325325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 36/77] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:13:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:13:44.331248 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3ce40e-0b03-4349-87f1-0ee4c2b3fefc None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:44.331945 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3ce40e-0b03-4349-87f1-0ee4c2b3fefc None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:44.332143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3ce40e-0b03-4349-87f1-0ee4c2b3fefc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:44.367322 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-2f3ce40e-0b03-4349-87f1-0ee4c2b3fefc None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:44.369402 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-2f3ce40e-0b03-4349-87f1-0ee4c2b3fefc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:44.509259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 37/78] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:13:44.511133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3ce40e-0b03-4349-87f1-0ee4c2b3fefc None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:44.514054 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f711f5-2334-455a-af31-9d9639bc997c None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:44.571538 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f28090b-7b81-4ad1-9f8a-9d2c525d7b79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:44.572387 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f28090b-7b81-4ad1-9f8a-9d2c525d7b79 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:44.572692 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f28090b-7b81-4ad1-9f8a-9d2c525d7b79 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:44.573100 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f28090b-7b81-4ad1-9f8a-9d2c525d7b79 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:44.573825 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f28090b-7b81-4ad1-9f8a-9d2c525d7b79 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:44.574137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f28090b-7b81-4ad1-9f8a-9d2c525d7b79 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:44.576080 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f28090b-7b81-4ad1-9f8a-9d2c525d7b79 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:44.591764 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 42/79] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:13:44 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:44.593798 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f28090b-7b81-4ad1-9f8a-9d2c525d7b79 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:44.636596 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:44.672115 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3ce40e-0b03-4349-87f1-0ee4c2b3fefc None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:44.705614 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:44.750117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc56f07e-b588-4591-b650-9ffc4ef3444e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:44.751650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc56f07e-b588-4591-b650-9ffc4ef3444e None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:44.752514 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc56f07e-b588-4591-b650-9ffc4ef3444e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:44.753338 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc56f07e-b588-4591-b650-9ffc4ef3444e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:44.861075 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f28090b-7b81-4ad1-9f8a-9d2c525d7b79 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:44.903949 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc56f07e-b588-4591-b650-9ffc4ef3444e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:44.907298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc56f07e-b588-4591-b650-9ffc4ef3444e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:44.961429 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdfda9e-d4fb-4405-a706-1b8e91cb4e42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:44.961835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdfda9e-d4fb-4405-a706-1b8e91cb4e42 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:44.962092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdfda9e-d4fb-4405-a706-1b8e91cb4e42 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:44.962372 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdfda9e-d4fb-4405-a706-1b8e91cb4e42 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:44.963083 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdfda9e-d4fb-4405-a706-1b8e91cb4e42 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:44.963394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdfda9e-d4fb-4405-a706-1b8e91cb4e42 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:44.969770 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdfda9e-d4fb-4405-a706-1b8e91cb4e42 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:44.982643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 43/80] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:13:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 390 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:44.984196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdfda9e-d4fb-4405-a706-1b8e91cb4e42 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:45.095289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:45.159518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 38/81] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:13:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 649 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:13:45.160161 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc56f07e-b588-4591-b650-9ffc4ef3444e None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:45.205653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdfda9e-d4fb-4405-a706-1b8e91cb4e42 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:45.210893 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:45.295674 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52348276-7f13-45c2-b75f-376cd8231ed0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:45.297504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52348276-7f13-45c2-b75f-376cd8231ed0 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:45.298663 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52348276-7f13-45c2-b75f-376cd8231ed0 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:45.299426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52348276-7f13-45c2-b75f-376cd8231ed0 None admin] PATH_INFO: `/v3/users/ac210aede44f454dbb9c3aa4a942ffab` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:45.300258 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc56f07e-b588-4591-b650-9ffc4ef3444e None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:45.301968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52348276-7f13-45c2-b75f-376cd8231ed0 None admin] RBAC: Authorizing `identity:get_user(user_id=ac210aede44f454dbb9c3aa4a942ffab)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:45.307029 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52348276-7f13-45c2-b75f-376cd8231ed0 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:45.310226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 44/82] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:13:44 2022] GET /identity/v3/users/ac210aede44f454dbb9c3aa4a942ffab => generated 275 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:45.313244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52348276-7f13-45c2-b75f-376cd8231ed0 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:45.358118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd28ed90-05b6-4641-88ea-40b6a6e1df24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:45.359192 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd28ed90-05b6-4641-88ea-40b6a6e1df24 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:45.359828 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd28ed90-05b6-4641-88ea-40b6a6e1df24 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:45.360349 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd28ed90-05b6-4641-88ea-40b6a6e1df24 None admin] PATH_INFO: `/v3/users/4fff265355db498da98d8cb52b62f573` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:45.361871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd28ed90-05b6-4641-88ea-40b6a6e1df24 None admin] RBAC: Authorizing `identity:get_user(user_id=4fff265355db498da98d8cb52b62f573)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:45.363847 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd28ed90-05b6-4641-88ea-40b6a6e1df24 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:45.366810 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 39/83] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:13:45 2022] GET /identity/v3/users/4fff265355db498da98d8cb52b62f573 => generated 247 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:45.367952 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd28ed90-05b6-4641-88ea-40b6a6e1df24 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:45.463816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:45.520757 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52348276-7f13-45c2-b75f-376cd8231ed0 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:45.571686 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd28ed90-05b6-4641-88ea-40b6a6e1df24 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:45.597841 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7269ab05-49ef-4489-b272-a90bae09baa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:45.598568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269ab05-49ef-4489-b272-a90bae09baa5 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:45.598955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269ab05-49ef-4489-b272-a90bae09baa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:45.599353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269ab05-49ef-4489-b272-a90bae09baa5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:45.616560 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7269ab05-49ef-4489-b272-a90bae09baa5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:13:45.623146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-7269ab05-49ef-4489-b272-a90bae09baa5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:13:45.623146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:13:45.623146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:13:45.623146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:13:45.623146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:13:45.623146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:13:45.623146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:13:45.623146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:13:45.623146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:13:45.623146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:13:45.623146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:13:45.623146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:13:45.623146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:13:45.623146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:13:45.623146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:13:45.623146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:13:45.624132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:13:45.624132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:13:45.624132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:13:45.624132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:13:45.624132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:13:45.624132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:13:45.624132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:13:45.624132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:13:45.624132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:13:45.624132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:13:45.624132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:13:45.624132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:13:45.624132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:13:45.624132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:13:45.624132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:13:45.624132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:13:45.625533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:13:45.625533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:13:45.625533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:13:45.625533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:13:45.625533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:13:45.625533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:13:45.625533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:13:45.625533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:13:45.625533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:13:45.625533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:13:45.625533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:13:45.625533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:13:45.627516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02afa7bf-d7c2-47e4-8e8e-636692e1b17b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:45.628109 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 45/84] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:13:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:13:45.633209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02afa7bf-d7c2-47e4-8e8e-636692e1b17b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:45.633209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02afa7bf-d7c2-47e4-8e8e-636692e1b17b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:45.633209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02afa7bf-d7c2-47e4-8e8e-636692e1b17b None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:45.633209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02afa7bf-d7c2-47e4-8e8e-636692e1b17b None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:45.634496 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02afa7bf-d7c2-47e4-8e8e-636692e1b17b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:45.634970 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7269ab05-49ef-4489-b272-a90bae09baa5 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:45.637377 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 40/85] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:13:45 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:45.638404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02afa7bf-d7c2-47e4-8e8e-636692e1b17b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:45.684724 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:45.770689 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02afa7bf-d7c2-47e4-8e8e-636692e1b17b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:45.786321 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:45.791307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7269ab05-49ef-4489-b272-a90bae09baa5 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:45.831731 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce35f40-6c67-4727-8fae-c95834d82da5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:45.832886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce35f40-6c67-4727-8fae-c95834d82da5 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:45.833330 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce35f40-6c67-4727-8fae-c95834d82da5 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:45.833816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce35f40-6c67-4727-8fae-c95834d82da5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:45.834422 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cacc7c-720f-483f-b348-a530f42d2834 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:45.835218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cacc7c-720f-483f-b348-a530f42d2834 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:45.835632 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cacc7c-720f-483f-b348-a530f42d2834 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:45.836070 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cacc7c-720f-483f-b348-a530f42d2834 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:45.949223 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cacc7c-720f-483f-b348-a530f42d2834 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:45.950935 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cacc7c-720f-483f-b348-a530f42d2834 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:45.974262 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce35f40-6c67-4727-8fae-c95834d82da5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:45.979080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce35f40-6c67-4727-8fae-c95834d82da5 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:45.990362 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:46.119956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 46/86] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:13:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 486 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:13:46.121183 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cacc7c-720f-483f-b348-a530f42d2834 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:46.210404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 41/87] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:13:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 572 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:13:46.211471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce35f40-6c67-4727-8fae-c95834d82da5 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:46.234023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:46.267053 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cacc7c-720f-483f-b348-a530f42d2834 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:46.330534 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:46.348013 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e82198b-6e60-4908-bf2f-7b4e08588f7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:46.350375 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e82198b-6e60-4908-bf2f-7b4e08588f7a None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:46.351157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e82198b-6e60-4908-bf2f-7b4e08588f7a None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:46.352198 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e82198b-6e60-4908-bf2f-7b4e08588f7a None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:46.354128 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e82198b-6e60-4908-bf2f-7b4e08588f7a None admin] RBAC: Adding query filter params (name=member) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:46.355676 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e82198b-6e60-4908-bf2f-7b4e08588f7a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:46.359861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e82198b-6e60-4908-bf2f-7b4e08588f7a None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:46.379102 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce35f40-6c67-4727-8fae-c95834d82da5 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:46.385853 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 47/88] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:13:46 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:46.387741 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e82198b-6e60-4908-bf2f-7b4e08588f7a None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:46.454399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681d5675-7d59-4a8c-9477-a4f8729fcee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:46.455668 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681d5675-7d59-4a8c-9477-a4f8729fcee2 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:46.456103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681d5675-7d59-4a8c-9477-a4f8729fcee2 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:46.456629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681d5675-7d59-4a8c-9477-a4f8729fcee2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:46.458016 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681d5675-7d59-4a8c-9477-a4f8729fcee2 None admin] RBAC: Adding query filter params (role.id=bc6aeaa495ca4f3dbe72905c744c2233, scope.project.id=957c97404bb9448e9d03eee0d3225b85, user.id=ac210aede44f454dbb9c3aa4a942ffab) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:46.458560 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681d5675-7d59-4a8c-9477-a4f8729fcee2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:46.461863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681d5675-7d59-4a8c-9477-a4f8729fcee2 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:46.478962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 42/89] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:13:46 2022] GET /identity/v3/role_assignments?user.id=ac210aede44f454dbb9c3aa4a942ffab&scope.project.id=957c97404bb9448e9d03eee0d3225b85&role.id=bc6aeaa495ca4f3dbe72905c744c2233 => generated 259 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:46.480659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681d5675-7d59-4a8c-9477-a4f8729fcee2 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:46.626379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e82198b-6e60-4908-bf2f-7b4e08588f7a None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:46.664967 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:46.731246 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ea3104-6150-43eb-9c57-07e3beea333f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:46.732426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ea3104-6150-43eb-9c57-07e3beea333f None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:46.732911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ea3104-6150-43eb-9c57-07e3beea333f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:46.733399 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ea3104-6150-43eb-9c57-07e3beea333f None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:46.740213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ea3104-6150-43eb-9c57-07e3beea333f None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:46.750852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ea3104-6150-43eb-9c57-07e3beea333f None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:46.754259 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 48/90] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:13:46 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 368 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:46.761252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ea3104-6150-43eb-9c57-07e3beea333f None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:46.779283 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681d5675-7d59-4a8c-9477-a4f8729fcee2 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:46.862643 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895f87d8-23d1-4349-840e-c19224bb78fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:46.864259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895f87d8-23d1-4349-840e-c19224bb78fd None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:46.865130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895f87d8-23d1-4349-840e-c19224bb78fd None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:46.866858 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895f87d8-23d1-4349-840e-c19224bb78fd None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:46.869790 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895f87d8-23d1-4349-840e-c19224bb78fd None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:46.872541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895f87d8-23d1-4349-840e-c19224bb78fd None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:46.876826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 43/91] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:13:46 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 397 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:46.879174 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895f87d8-23d1-4349-840e-c19224bb78fd None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:46.993016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:47.104493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ea3104-6150-43eb-9c57-07e3beea333f None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:47.144277 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895f87d8-23d1-4349-840e-c19224bb78fd None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:47.175504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2785d9fe-7bad-4dce-8b24-3afc1b8820a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:47.176567 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2785d9fe-7bad-4dce-8b24-3afc1b8820a4 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:47.177075 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2785d9fe-7bad-4dce-8b24-3afc1b8820a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:47.177576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2785d9fe-7bad-4dce-8b24-3afc1b8820a4 None admin] PATH_INFO: `/v3/users/ac210aede44f454dbb9c3aa4a942ffab` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:47.179915 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2785d9fe-7bad-4dce-8b24-3afc1b8820a4 None admin] RBAC: Authorizing `identity:get_user(user_id=ac210aede44f454dbb9c3aa4a942ffab)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:47.181957 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2785d9fe-7bad-4dce-8b24-3afc1b8820a4 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:47.184300 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 49/92] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:13:46 2022] GET /identity/v3/users/ac210aede44f454dbb9c3aa4a942ffab => generated 275 bytes in 429 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:47.185428 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2785d9fe-7bad-4dce-8b24-3afc1b8820a4 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:47.250819 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcdc02c-a938-4945-9dd2-fb8f016b4d18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:47.251479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcdc02c-a938-4945-9dd2-fb8f016b4d18 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:47.251831 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcdc02c-a938-4945-9dd2-fb8f016b4d18 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:47.252184 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcdc02c-a938-4945-9dd2-fb8f016b4d18 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:47.253041 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcdc02c-a938-4945-9dd2-fb8f016b4d18 None admin] RBAC: Adding query filter params (role.id=e114d28e2dd147fcb39452dd461791c0, scope.project.id=957c97404bb9448e9d03eee0d3225b85, user.id=ac210aede44f454dbb9c3aa4a942ffab) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:47.253422 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcdc02c-a938-4945-9dd2-fb8f016b4d18 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:47.255401 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcdc02c-a938-4945-9dd2-fb8f016b4d18 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:47.272365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 44/93] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:13:46 2022] GET /identity/v3/role_assignments?user.id=ac210aede44f454dbb9c3aa4a942ffab&scope.project.id=957c97404bb9448e9d03eee0d3225b85&role.id=e114d28e2dd147fcb39452dd461791c0 => generated 259 bytes in 386 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:47.272365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcdc02c-a938-4945-9dd2-fb8f016b4d18 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:47.275543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:47.395600 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2785d9fe-7bad-4dce-8b24-3afc1b8820a4 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:47.404881 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcdc02c-a938-4945-9dd2-fb8f016b4d18 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:47.451801 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c6d090-930b-4026-af34-e4283722c833 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:47.452730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c6d090-930b-4026-af34-e4283722c833 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:47.453451 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c6d090-930b-4026-af34-e4283722c833 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:47.454123 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c6d090-930b-4026-af34-e4283722c833 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:47.504600 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:47.547006 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c6d090-930b-4026-af34-e4283722c833 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:47.549576 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c6d090-930b-4026-af34-e4283722c833 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:47.558102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0d3698-eba7-4dd6-b7fc-4cce0e505c19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:47.559162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0d3698-eba7-4dd6-b7fc-4cce0e505c19 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:47.559661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0d3698-eba7-4dd6-b7fc-4cce0e505c19 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:47.560160 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0d3698-eba7-4dd6-b7fc-4cce0e505c19 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:47.561662 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0d3698-eba7-4dd6-b7fc-4cce0e505c19 None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=957c97404bb9448e9d03eee0d3225b85, user.id=4fff265355db498da98d8cb52b62f573) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:47.562165 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:47.562455 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0d3698-eba7-4dd6-b7fc-4cce0e505c19 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:47.567829 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0d3698-eba7-4dd6-b7fc-4cce0e505c19 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:47.580797 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 50/94] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:13:47 2022] GET /identity/v3/role_assignments?user.id=4fff265355db498da98d8cb52b62f573&scope.project.id=957c97404bb9448e9d03eee0d3225b85&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 259 bytes in 395 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:47.582978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bf433e-6653-421c-a599-fa61e246dac6 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:47.583487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bf433e-6653-421c-a599-fa61e246dac6 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:47.583984 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bf433e-6653-421c-a599-fa61e246dac6 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:47.586999 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 51/95] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:47 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:47.595873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f65ca38-e703-4195-9a5d-5e1e090a0ba3 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:47.596429 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f65ca38-e703-4195-9a5d-5e1e090a0ba3 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:47.596919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f65ca38-e703-4195-9a5d-5e1e090a0ba3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:47.647645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-8f65ca38-e703-4195-9a5d-5e1e090a0ba3 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:47.649709 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8f65ca38-e703-4195-9a5d-5e1e090a0ba3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:47.686711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 45/96] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:13:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:13:47.692767 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c6d090-930b-4026-af34-e4283722c833 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:47.823927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 52/97] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:13:47.832965 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c6d090-930b-4026-af34-e4283722c833 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:47.844769 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:47.902395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec149bfe-e057-42b9-a885-4ca4638893ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:47.903072 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec149bfe-e057-42b9-a885-4ca4638893ad None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:47.903353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec149bfe-e057-42b9-a885-4ca4638893ad None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:47.903635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec149bfe-e057-42b9-a885-4ca4638893ad None admin] PATH_INFO: `/v3/projects/cdc2e463faa84c7fb617e11187fa0a6c` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:47.905089 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec149bfe-e057-42b9-a885-4ca4638893ad None admin] RBAC: Authorizing `identity:get_project(project_id=cdc2e463faa84c7fb617e11187fa0a6c)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:47.906626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec149bfe-e057-42b9-a885-4ca4638893ad None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:47.908854 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 46/98] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:13:47 2022] GET /identity/v3/projects/cdc2e463faa84c7fb617e11187fa0a6c => generated 312 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:13:47.916514 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f65ca38-e703-4195-9a5d-5e1e090a0ba3 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:48.195494 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f65ca38-e703-4195-9a5d-5e1e090a0ba3 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:48.231689 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:48.246803 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca24f96-e1bc-4f3e-be7f-87084357ea92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:48.247441 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca24f96-e1bc-4f3e-be7f-87084357ea92 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:48.247727 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca24f96-e1bc-4f3e-be7f-87084357ea92 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:48.248008 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca24f96-e1bc-4f3e-be7f-87084357ea92 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:48.248977 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca24f96-e1bc-4f3e-be7f-87084357ea92 None admin] RBAC: Adding query filter params (role.id=bc6aeaa495ca4f3dbe72905c744c2233, scope.project.id=cdc2e463faa84c7fb617e11187fa0a6c, user.id=ac210aede44f454dbb9c3aa4a942ffab) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:48.249360 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca24f96-e1bc-4f3e-be7f-87084357ea92 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:48.250933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca24f96-e1bc-4f3e-be7f-87084357ea92 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:48.261869 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 53/99] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:13:47 2022] GET /identity/v3/role_assignments?user.id=ac210aede44f454dbb9c3aa4a942ffab&scope.project.id=cdc2e463faa84c7fb617e11187fa0a6c&role.id=bc6aeaa495ca4f3dbe72905c744c2233 => generated 259 bytes in 347 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:48.406682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69eb0521-df99-46e8-a302-242948aef64b None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:48.407850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69eb0521-df99-46e8-a302-242948aef64b None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:48.408789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69eb0521-df99-46e8-a302-242948aef64b None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:48.411739 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 47/100] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:48 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:48.421071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe06b90-8a42-47c2-992e-8fa53da4d7fb None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:48.421515 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe06b90-8a42-47c2-992e-8fa53da4d7fb None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:48.422006 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe06b90-8a42-47c2-992e-8fa53da4d7fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:48.468171 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-bfe06b90-8a42-47c2-992e-8fa53da4d7fb None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:48.470140 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-bfe06b90-8a42-47c2-992e-8fa53da4d7fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:48.624253 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 54/101] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:13:48.858003 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69eb0521-df99-46e8-a302-242948aef64b None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:48.993493 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69eb0521-df99-46e8-a302-242948aef64b None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:49.038752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea0441f-eb53-4b50-ba55-afa6bb74cf3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:49.039828 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea0441f-eb53-4b50-ba55-afa6bb74cf3d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:49.040372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea0441f-eb53-4b50-ba55-afa6bb74cf3d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:49.040867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea0441f-eb53-4b50-ba55-afa6bb74cf3d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:49.042430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea0441f-eb53-4b50-ba55-afa6bb74cf3d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:49.044268 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea0441f-eb53-4b50-ba55-afa6bb74cf3d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:49.046309 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 48/102] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:13:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:49.056714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe06b90-8a42-47c2-992e-8fa53da4d7fb None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:49.284378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe06b90-8a42-47c2-992e-8fa53da4d7fb None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:49.322532 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:49.366956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09e23f0-40d2-40c7-b514-1a6b0dc86dc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:49.368014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09e23f0-40d2-40c7-b514-1a6b0dc86dc8 None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:49.368552 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09e23f0-40d2-40c7-b514-1a6b0dc86dc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:49.369047 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09e23f0-40d2-40c7-b514-1a6b0dc86dc8 None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:49.370303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09e23f0-40d2-40c7-b514-1a6b0dc86dc8 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:49.373141 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09e23f0-40d2-40c7-b514-1a6b0dc86dc8 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:49.408915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea0441f-eb53-4b50-ba55-afa6bb74cf3d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:49.445371 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 55/103] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Oct 4 14:13:49 2022] POST /identity/v3/projects => generated 302 bytes in 390 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:13:49.544454 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:49.691654 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea0441f-eb53-4b50-ba55-afa6bb74cf3d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:49.775034 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51c82c4-0e73-4edc-ba4d-2f17eb99ea9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:49.775703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51c82c4-0e73-4edc-ba4d-2f17eb99ea9b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:49.775990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51c82c4-0e73-4edc-ba4d-2f17eb99ea9b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:49.776267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51c82c4-0e73-4edc-ba4d-2f17eb99ea9b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:49.795186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51c82c4-0e73-4edc-ba4d-2f17eb99ea9b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:13:49.800404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-e51c82c4-0e73-4edc-ba4d-2f17eb99ea9b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:13:49.800404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:13:49.800404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:13:49.800404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:13:49.800404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:13:49.800404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:13:49.800404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:13:49.800404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:13:49.800404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:13:49.800404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:13:49.800404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:13:49.800404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:13:49.800404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:13:49.800404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:13:49.800404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:13:49.800404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:13:49.801734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:13:49.801734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:13:49.801734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:13:49.801734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:13:49.801734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:13:49.801734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:13:49.801734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:13:49.801734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:13:49.801734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:13:49.801734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:13:49.801734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:13:49.801734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:13:49.801734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:13:49.801734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:13:49.801734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:13:49.801734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:13:49.803651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:13:49.803651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:13:49.803651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:13:49.803651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:13:49.803651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:13:49.803651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:13:49.803651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:13:49.803651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:13:49.803651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:13:49.803651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:13:49.803651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:13:49.803651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:13:49.803651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 49/104] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:13:49 2022] GET /identity/v3/roles/reader => generated 84 bytes in 394 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:13:49.814891 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09e23f0-40d2-40c7-b514-1a6b0dc86dc8 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:49.964011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9b33d5-7680-4b78-b5d9-64699037d0b9 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:49.964463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9b33d5-7680-4b78-b5d9-64699037d0b9 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:49.964694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9b33d5-7680-4b78-b5d9-64699037d0b9 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:49.966627 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 50/105] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:13:49.973165 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f040bb-91ae-4bde-961c-d7697ecb77bf None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:49.973550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f040bb-91ae-4bde-961c-d7697ecb77bf None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:49.973807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f040bb-91ae-4bde-961c-d7697ecb77bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:50.011478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-07f040bb-91ae-4bde-961c-d7697ecb77bf None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:50.012799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-07f040bb-91ae-4bde-961c-d7697ecb77bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:50.093294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09e23f0-40d2-40c7-b514-1a6b0dc86dc8 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:50.149343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 51/106] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:13:50.171225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662b8e11-c3fa-4db9-bb37-1a5313db5129 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:50.171874 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662b8e11-c3fa-4db9-bb37-1a5313db5129 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:50.172167 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662b8e11-c3fa-4db9-bb37-1a5313db5129 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:50.172472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662b8e11-c3fa-4db9-bb37-1a5313db5129 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:50.173178 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662b8e11-c3fa-4db9-bb37-1a5313db5129 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:50.173491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662b8e11-c3fa-4db9-bb37-1a5313db5129 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:50.174933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662b8e11-c3fa-4db9-bb37-1a5313db5129 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:50.186423 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 56/107] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:13:49 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 373 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:50.195476 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f040bb-91ae-4bde-961c-d7697ecb77bf None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:50.241443 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:50.344151 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f040bb-91ae-4bde-961c-d7697ecb77bf None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:50.400085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa877c58-1198-4d2b-be07-ed0769390030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:50.400791 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa877c58-1198-4d2b-be07-ed0769390030 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:50.401072 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa877c58-1198-4d2b-be07-ed0769390030 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:50.401348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa877c58-1198-4d2b-be07-ed0769390030 None admin] PATH_INFO: `/v3/users/229bce563b8c45fb8a24a170a209bf2f` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:50.424673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa877c58-1198-4d2b-be07-ed0769390030 None admin] RBAC: Authorizing `identity:get_user(user_id=229bce563b8c45fb8a24a170a209bf2f)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:50.426218 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa877c58-1198-4d2b-be07-ed0769390030 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:50.428456 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 52/108] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:13:50 2022] GET /identity/v3/users/229bce563b8c45fb8a24a170a209bf2f => generated 289 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:50.437080 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662b8e11-c3fa-4db9-bb37-1a5313db5129 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:50.486887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:50.590783 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662b8e11-c3fa-4db9-bb37-1a5313db5129 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:50.643247 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4c7a50-f4b9-4e00-8d6d-62eb8d6a5be0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:50.643890 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4c7a50-f4b9-4e00-8d6d-62eb8d6a5be0 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:50.644160 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4c7a50-f4b9-4e00-8d6d-62eb8d6a5be0 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:50.644476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4c7a50-f4b9-4e00-8d6d-62eb8d6a5be0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:50.737102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4c7a50-f4b9-4e00-8d6d-62eb8d6a5be0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:50.738645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4c7a50-f4b9-4e00-8d6d-62eb8d6a5be0 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:50.776954 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:50.936902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 57/109] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:13:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 500 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 14:13:50.944811 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa877c58-1198-4d2b-be07-ed0769390030 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:51.011243 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4c7a50-f4b9-4e00-8d6d-62eb8d6a5be0 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:51.206073 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:51.224346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa877c58-1198-4d2b-be07-ed0769390030 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:51.355330 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ec0a47-5472-426b-92c1-648f30824812 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:51.356480 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ec0a47-5472-426b-92c1-648f30824812 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:51.356988 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ec0a47-5472-426b-92c1-648f30824812 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:51.358576 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ec0a47-5472-426b-92c1-648f30824812 None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:51.366637 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ec0a47-5472-426b-92c1-648f30824812 None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:51.369522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ec0a47-5472-426b-92c1-648f30824812 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:51.373353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 53/110] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:13:50 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 430 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:13:51.387140 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4c7a50-f4b9-4e00-8d6d-62eb8d6a5be0 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:51.389205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ec0a47-5472-426b-92c1-648f30824812 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:51.505302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:51.534060 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a42fea-6561-40ee-a70b-2bee931c6df3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:51.534747 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a42fea-6561-40ee-a70b-2bee931c6df3 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:51.535036 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a42fea-6561-40ee-a70b-2bee931c6df3 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:51.535315 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a42fea-6561-40ee-a70b-2bee931c6df3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:51.573081 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a42fea-6561-40ee-a70b-2bee931c6df3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:13:51.578185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-a3a42fea-6561-40ee-a70b-2bee931c6df3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:13:51.578185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:13:51.578185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:13:51.578185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:13:51.578185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:13:51.578185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:13:51.578185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:13:51.578185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:13:51.578185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:13:51.578185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:13:51.578185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:13:51.578185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:13:51.578185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:13:51.578185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:13:51.578185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:13:51.578185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:13:51.579942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:13:51.579942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:13:51.579942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:13:51.579942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:13:51.579942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:13:51.579942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:13:51.579942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:13:51.579942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:13:51.579942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:13:51.579942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:13:51.579942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:13:51.579942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:13:51.579942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:13:51.579942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:13:51.579942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:13:51.579942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:13:51.583224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:13:51.583224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:13:51.583224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:13:51.583224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:13:51.583224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:13:51.583224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:13:51.583224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:13:51.583224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:13:51.583224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:13:51.583224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:13:51.583224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:13:51.583224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:13:51.583224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 58/111] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:13:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 569 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:13:51.592471 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a42fea-6561-40ee-a70b-2bee931c6df3 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:51.641576 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ec0a47-5472-426b-92c1-648f30824812 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:51.714481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27af80ea-b684-4c7c-9846-cf69dc8f700d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:51.715008 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27af80ea-b684-4c7c-9846-cf69dc8f700d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:51.715288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27af80ea-b684-4c7c-9846-cf69dc8f700d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:51.715563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27af80ea-b684-4c7c-9846-cf69dc8f700d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:51.716338 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27af80ea-b684-4c7c-9846-cf69dc8f700d None admin] RBAC: Adding query filter params (role.id=f425b98e3acf4e13b0da24b21e59cfce, scope.project.id=957c97404bb9448e9d03eee0d3225b85, user.id=229bce563b8c45fb8a24a170a209bf2f) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:51.716649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27af80ea-b684-4c7c-9846-cf69dc8f700d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:51.718114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27af80ea-b684-4c7c-9846-cf69dc8f700d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:51.738617 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 54/112] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:13:51 2022] GET /identity/v3/role_assignments?user.id=229bce563b8c45fb8a24a170a209bf2f&scope.project.id=957c97404bb9448e9d03eee0d3225b85&role.id=f425b98e3acf4e13b0da24b21e59cfce => generated 259 bytes in 351 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:51.832203 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a42fea-6561-40ee-a70b-2bee931c6df3 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:51.886351 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07e8de7-9e87-4e5e-afcc-ba0827296c57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:51.887009 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07e8de7-9e87-4e5e-afcc-ba0827296c57 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:51.887284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07e8de7-9e87-4e5e-afcc-ba0827296c57 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:51.887565 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07e8de7-9e87-4e5e-afcc-ba0827296c57 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:51.888273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07e8de7-9e87-4e5e-afcc-ba0827296c57 None admin] RBAC: Adding query filter params (name=member) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:51.888615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07e8de7-9e87-4e5e-afcc-ba0827296c57 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:51.890067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07e8de7-9e87-4e5e-afcc-ba0827296c57 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:51.904623 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 59/113] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:13:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:51.911123 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27af80ea-b684-4c7c-9846-cf69dc8f700d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:51.950608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:52.063363 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27af80ea-b684-4c7c-9846-cf69dc8f700d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:52.116539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11493dc-0d01-41f9-af13-8ba7b838272c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:52.117212 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11493dc-0d01-41f9-af13-8ba7b838272c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:52.117495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11493dc-0d01-41f9-af13-8ba7b838272c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:52.117775 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11493dc-0d01-41f9-af13-8ba7b838272c None admin] PATH_INFO: `/v3/users/ac210aede44f454dbb9c3aa4a942ffab` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:52.125393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11493dc-0d01-41f9-af13-8ba7b838272c None admin] RBAC: Authorizing `identity:get_user(user_id=ac210aede44f454dbb9c3aa4a942ffab)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:52.127192 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11493dc-0d01-41f9-af13-8ba7b838272c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:52.129872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 55/114] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:13:51 2022] GET /identity/v3/users/ac210aede44f454dbb9c3aa4a942ffab => generated 275 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:52.136062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07e8de7-9e87-4e5e-afcc-ba0827296c57 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:52.223184 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:52.299665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2db504-2a41-444a-aafd-91db00c0535f None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:52.300078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2db504-2a41-444a-aafd-91db00c0535f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:52.300220 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2db504-2a41-444a-aafd-91db00c0535f None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:52.302081 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 56/115] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:52.310335 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d60f54-46ff-4ade-9324-5fd4d2b8784d None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:52.310601 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d60f54-46ff-4ade-9324-5fd4d2b8784d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:52.310931 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d60f54-46ff-4ade-9324-5fd4d2b8784d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:52.329736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07e8de7-9e87-4e5e-afcc-ba0827296c57 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:52.348189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-76d60f54-46ff-4ade-9324-5fd4d2b8784d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:52.349630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-76d60f54-46ff-4ade-9324-5fd4d2b8784d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:52.410631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3924fa37-5ca2-4b1a-9d5f-536c667d16f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:52.411482 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3924fa37-5ca2-4b1a-9d5f-536c667d16f7 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:52.411611 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3924fa37-5ca2-4b1a-9d5f-536c667d16f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:52.412090 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3924fa37-5ca2-4b1a-9d5f-536c667d16f7 None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:52.414111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3924fa37-5ca2-4b1a-9d5f-536c667d16f7 None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:52.415628 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3924fa37-5ca2-4b1a-9d5f-536c667d16f7 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:52.424366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 60/116] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:13:52 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:52.429212 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3924fa37-5ca2-4b1a-9d5f-536c667d16f7 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:52.513116 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 57/117] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:13:52.707694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49512c37-6bde-4c6c-8ab7-7112e9e07528 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:52.708670 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49512c37-6bde-4c6c-8ab7-7112e9e07528 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:52.708670 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49512c37-6bde-4c6c-8ab7-7112e9e07528 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:52.711579 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 58/118] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:52.727535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3924fa37-5ca2-4b1a-9d5f-536c667d16f7 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:52.732745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd0293-c173-4cc2-9c40-9eb6c5612966 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:52.733077 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd0293-c173-4cc2-9c40-9eb6c5612966 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:52.733321 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd0293-c173-4cc2-9c40-9eb6c5612966 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:52.772663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-87fd0293-c173-4cc2-9c40-9eb6c5612966 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:52.775526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-87fd0293-c173-4cc2-9c40-9eb6c5612966 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:52.796966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:52.808644 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:52.857906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a878a34-cbed-4dc4-9799-2f81b53656bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:52.859602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a878a34-cbed-4dc4-9799-2f81b53656bd None admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:52.860458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a878a34-cbed-4dc4-9799-2f81b53656bd None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:52.861253 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a878a34-cbed-4dc4-9799-2f81b53656bd None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85/users/ac210aede44f454dbb9c3aa4a942ffab/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:52.871290 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a878a34-cbed-4dc4-9799-2f81b53656bd None admin] RBAC: Authorizing `identity:create_grant(project_id=957c97404bb9448e9d03eee0d3225b85, user_id=ac210aede44f454dbb9c3aa4a942ffab, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:52.874885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a878a34-cbed-4dc4-9799-2f81b53656bd None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:52.898916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 59/119] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:13:52.918232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 61/120] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:13:52 2022] PUT /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85/users/ac210aede44f454dbb9c3aa4a942ffab/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 492 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:13:52.927701 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fd0293-c173-4cc2-9c40-9eb6c5612966 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:53.050549 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fd0293-c173-4cc2-9c40-9eb6c5612966 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:53.096339 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1528b2-9a32-494f-8159-098085ed9d6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:53.097003 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1528b2-9a32-494f-8159-098085ed9d6f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:53.097290 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1528b2-9a32-494f-8159-098085ed9d6f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:53.097580 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1528b2-9a32-494f-8159-098085ed9d6f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:53.123233 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1528b2-9a32-494f-8159-098085ed9d6f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:13:53.128381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-4e1528b2-9a32-494f-8159-098085ed9d6f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:13:53.128381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:13:53.128381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:13:53.128381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:13:53.128381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:13:53.128381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:13:53.128381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:13:53.128381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:13:53.128381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:13:53.128381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:13:53.128381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:13:53.128381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:13:53.128381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:13:53.128381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:13:53.128381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:13:53.128381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:13:53.130017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:13:53.130017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:13:53.130017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:13:53.130017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:13:53.130017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:13:53.130017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:13:53.130017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:13:53.130017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:13:53.130017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:13:53.130017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:13:53.130017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:13:53.130017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:13:53.130017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:13:53.130017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:13:53.130017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:13:53.130017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:13:53.131574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:13:53.131574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:13:53.131574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:13:53.131574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:13:53.131574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:13:53.131574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:13:53.131574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:13:53.131574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:13:53.131574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:13:53.131574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:13:53.131574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:13:53.131574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:13:53.131574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 60/121] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:13:52 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:13:53.136515 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a878a34-cbed-4dc4-9799-2f81b53656bd None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:53.207024 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf462e8-ff22-4cf6-b2dd-f47ff796a354 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:53.207426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf462e8-ff22-4cf6-b2dd-f47ff796a354 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:53.207576 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf462e8-ff22-4cf6-b2dd-f47ff796a354 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:53.209618 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 61/122] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:13:53.221878 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fde5d82-6ff4-49b3-80a9-4e94431c14d4 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:53.222290 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fde5d82-6ff4-49b3-80a9-4e94431c14d4 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:53.222532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fde5d82-6ff4-49b3-80a9-4e94431c14d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:53.274530 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-4fde5d82-6ff4-49b3-80a9-4e94431c14d4 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:53.277224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-4fde5d82-6ff4-49b3-80a9-4e94431c14d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:53.298077 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:53.500363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a878a34-cbed-4dc4-9799-2f81b53656bd None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:53.517907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 62/123] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 297 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:13:53.521655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fde5d82-6ff4-49b3-80a9-4e94431c14d4 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:53.563344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c9c075-577c-448d-8772-e83bf44003e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:53.563992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c9c075-577c-448d-8772-e83bf44003e1 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:53.564280 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c9c075-577c-448d-8772-e83bf44003e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:53.564609 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c9c075-577c-448d-8772-e83bf44003e1 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:53.565331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c9c075-577c-448d-8772-e83bf44003e1 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:53.565670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c9c075-577c-448d-8772-e83bf44003e1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:53.567117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c9c075-577c-448d-8772-e83bf44003e1 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:53.586803 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 62/124] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:13:53 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 451 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:53.592755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c9c075-577c-448d-8772-e83bf44003e1 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:53.670188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:53.799812 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c9c075-577c-448d-8772-e83bf44003e1 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:53.875588 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0b733a-4b61-4764-9968-d55251d56e2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:53.876256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0b733a-4b61-4764-9968-d55251d56e2e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:53.876579 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0b733a-4b61-4764-9968-d55251d56e2e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:53.877017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0b733a-4b61-4764-9968-d55251d56e2e None admin] PATH_INFO: `/v3/users/ac210aede44f454dbb9c3aa4a942ffab` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:53.878073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0b733a-4b61-4764-9968-d55251d56e2e None admin] RBAC: Authorizing `identity:get_user(user_id=ac210aede44f454dbb9c3aa4a942ffab)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:53.879654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0b733a-4b61-4764-9968-d55251d56e2e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:53.881739 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 63/125] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:13:53 2022] GET /identity/v3/users/ac210aede44f454dbb9c3aa4a942ffab => generated 275 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:13:53.887294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0b733a-4b61-4764-9968-d55251d56e2e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:53.889752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fde5d82-6ff4-49b3-80a9-4e94431c14d4 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:54.018800 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0acd7c-f209-4ebd-b44f-893bc9a45e7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:54.019999 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0acd7c-f209-4ebd-b44f-893bc9a45e7a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:54.020529 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0acd7c-f209-4ebd-b44f-893bc9a45e7a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:54.021017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0acd7c-f209-4ebd-b44f-893bc9a45e7a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:54.049749 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:54.064841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0acd7c-f209-4ebd-b44f-893bc9a45e7a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:13:54.078375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-6d0acd7c-f209-4ebd-b44f-893bc9a45e7a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:13:54.078375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:13:54.078375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:13:54.078375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:13:54.078375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:13:54.078375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:13:54.078375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:13:54.078375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:13:54.078375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:13:54.078375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:13:54.078375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:13:54.078375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:13:54.078375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:13:54.078375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:13:54.078375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:13:54.078375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:13:54.080552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:13:54.080552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:13:54.080552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:13:54.080552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:13:54.080552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:13:54.080552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:13:54.080552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:13:54.080552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:13:54.080552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:13:54.080552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:13:54.080552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:13:54.080552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:13:54.080552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:13:54.080552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:13:54.080552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:13:54.080552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:13:54.082253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:13:54.082253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:13:54.082253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:13:54.082253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:13:54.082253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:13:54.082253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:13:54.082253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:13:54.082253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:13:54.082253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:13:54.082253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:13:54.082253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:13:54.082253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:13:54.092710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 63/126] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:13:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 570 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:13:54.092710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0acd7c-f209-4ebd-b44f-893bc9a45e7a None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:54.175319 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0b733a-4b61-4764-9968-d55251d56e2e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:54.205155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:54.241960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553f58dd-dc10-429c-864e-dd52daa42fff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:54.242812 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553f58dd-dc10-429c-864e-dd52daa42fff None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:54.243161 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553f58dd-dc10-429c-864e-dd52daa42fff None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:54.243502 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553f58dd-dc10-429c-864e-dd52daa42fff None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:54.244947 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553f58dd-dc10-429c-864e-dd52daa42fff None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:54.246755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553f58dd-dc10-429c-864e-dd52daa42fff None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:54.253782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 64/127] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:13:53 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 362 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:54.255454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c74ca8-f276-4cee-981b-85f741a4600a None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:54.255914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c74ca8-f276-4cee-981b-85f741a4600a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:54.256363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c74ca8-f276-4cee-981b-85f741a4600a None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:54.258306 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 65/128] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:54.259380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c74ca8-f276-4cee-981b-85f741a4600a None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:54.267285 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0acd7c-f209-4ebd-b44f-893bc9a45e7a None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:54.342159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e231ebe1-d7af-43d8-9565-4dde283b579c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:54.343744 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e231ebe1-d7af-43d8-9565-4dde283b579c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:54.344560 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e231ebe1-d7af-43d8-9565-4dde283b579c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:54.345287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e231ebe1-d7af-43d8-9565-4dde283b579c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:54.369673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e231ebe1-d7af-43d8-9565-4dde283b579c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:13:54.380575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-e231ebe1-d7af-43d8-9565-4dde283b579c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:13:54.380575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:13:54.380575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:13:54.380575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:13:54.380575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:13:54.380575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:13:54.380575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:13:54.380575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:13:54.380575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:13:54.380575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:13:54.380575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:13:54.380575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:13:54.380575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:13:54.380575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:13:54.380575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:13:54.380575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:13:54.381758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:13:54.381758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:13:54.381758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:13:54.381758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:13:54.381758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:13:54.381758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:13:54.381758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:13:54.381758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:13:54.381758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:13:54.381758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:13:54.381758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:13:54.381758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:13:54.381758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:13:54.381758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:13:54.381758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:13:54.381758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:13:54.382636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:13:54.382636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:13:54.382636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:13:54.382636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:13:54.382636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:13:54.382636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:13:54.382636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:13:54.382636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:13:54.382636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:13:54.382636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:13:54.382636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:13:54.382636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:13:54.385280 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 64/129] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:13:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:13:54.386405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e231ebe1-d7af-43d8-9565-4dde283b579c None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:54.400299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c74ca8-f276-4cee-981b-85f741a4600a None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:54.464864 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:54.471920 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef006ab-ea23-4d0f-aa6c-722a663c79a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:54.473402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef006ab-ea23-4d0f-aa6c-722a663c79a7 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:54.473862 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef006ab-ea23-4d0f-aa6c-722a663c79a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:54.475012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef006ab-ea23-4d0f-aa6c-722a663c79a7 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:54.476556 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef006ab-ea23-4d0f-aa6c-722a663c79a7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:54.476860 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef006ab-ea23-4d0f-aa6c-722a663c79a7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:54.479685 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef006ab-ea23-4d0f-aa6c-722a663c79a7 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:54.502124 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 66/130] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:13:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:54.507355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e118237-f70d-4690-a27e-ed8e7b6617e8 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:54.508120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e118237-f70d-4690-a27e-ed8e7b6617e8 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:54.508647 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e118237-f70d-4690-a27e-ed8e7b6617e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:54.520122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e231ebe1-d7af-43d8-9565-4dde283b579c None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:54.542954 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-8e118237-f70d-4690-a27e-ed8e7b6617e8 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:54.545174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8e118237-f70d-4690-a27e-ed8e7b6617e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:54.562865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce46d4d-38a9-452c-9789-254501f11d7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:54.563885 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce46d4d-38a9-452c-9789-254501f11d7f None admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:54.564381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce46d4d-38a9-452c-9789-254501f11d7f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:54.564860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce46d4d-38a9-452c-9789-254501f11d7f None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85/users/ac210aede44f454dbb9c3aa4a942ffab/roles/e114d28e2dd147fcb39452dd461791c0` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:54.567459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce46d4d-38a9-452c-9789-254501f11d7f None admin] RBAC: Authorizing `identity:create_grant(project_id=957c97404bb9448e9d03eee0d3225b85, user_id=ac210aede44f454dbb9c3aa4a942ffab, role_id=e114d28e2dd147fcb39452dd461791c0)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:54.569559 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce46d4d-38a9-452c-9789-254501f11d7f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:54.589041 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 65/131] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:13:54 2022] PUT /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85/users/ac210aede44f454dbb9c3aa4a942ffab/roles/e114d28e2dd147fcb39452dd461791c0 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:13:54.590347 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce46d4d-38a9-452c-9789-254501f11d7f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:54.659266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:54.731527 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce46d4d-38a9-452c-9789-254501f11d7f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:54.768784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:54.778860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3c3f11-b18c-49a5-bb85-f7fa973451e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:54.779811 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3c3f11-b18c-49a5-bb85-f7fa973451e4 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:54.780336 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3c3f11-b18c-49a5-bb85-f7fa973451e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:54.781065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3c3f11-b18c-49a5-bb85-f7fa973451e4 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:54.781981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3c3f11-b18c-49a5-bb85-f7fa973451e4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:54.782581 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3c3f11-b18c-49a5-bb85-f7fa973451e4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:54.784568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3c3f11-b18c-49a5-bb85-f7fa973451e4 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:54.797656 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 66/132] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:13:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:54.798959 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3c3f11-b18c-49a5-bb85-f7fa973451e4 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:54.827924 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 67/133] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 325 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:13:54.829452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e118237-f70d-4690-a27e-ed8e7b6617e8 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:54.934457 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3c3f11-b18c-49a5-bb85-f7fa973451e4 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:54.987789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa9d1c5-d909-432c-bae5-95fd79fa2bfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:54.988954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa9d1c5-d909-432c-bae5-95fd79fa2bfc None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:54.989438 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa9d1c5-d909-432c-bae5-95fd79fa2bfc None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:54.990000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa9d1c5-d909-432c-bae5-95fd79fa2bfc None admin] PATH_INFO: `/v3/users/4fff265355db498da98d8cb52b62f573` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:54.992050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa9d1c5-d909-432c-bae5-95fd79fa2bfc None admin] RBAC: Authorizing `identity:get_user(user_id=4fff265355db498da98d8cb52b62f573)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:54.993926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa9d1c5-d909-432c-bae5-95fd79fa2bfc None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:54.995944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 67/134] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:13:54 2022] GET /identity/v3/users/4fff265355db498da98d8cb52b62f573 => generated 247 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:55.012358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa9d1c5-d909-432c-bae5-95fd79fa2bfc None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:55.092787 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:55.143679 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa9d1c5-d909-432c-bae5-95fd79fa2bfc None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:55.170117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e118237-f70d-4690-a27e-ed8e7b6617e8 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:55.186391 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58904907-c3f0-454c-ba4c-21c8b73a9b6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:55.187372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58904907-c3f0-454c-ba4c-21c8b73a9b6a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:55.187856 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58904907-c3f0-454c-ba4c-21c8b73a9b6a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:55.188359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58904907-c3f0-454c-ba4c-21c8b73a9b6a None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:55.189942 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58904907-c3f0-454c-ba4c-21c8b73a9b6a None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:55.191990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58904907-c3f0-454c-ba4c-21c8b73a9b6a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:55.194644 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 68/135] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:13:55 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:13:55.195913 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58904907-c3f0-454c-ba4c-21c8b73a9b6a None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:55.249239 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84d1388-1cf9-4a6c-b02d-6cd928ec13b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:55.250291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84d1388-1cf9-4a6c-b02d-6cd928ec13b8 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:55.250780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84d1388-1cf9-4a6c-b02d-6cd928ec13b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:55.261086 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84d1388-1cf9-4a6c-b02d-6cd928ec13b8 None admin] PATH_INFO: `/v3/users/ac210aede44f454dbb9c3aa4a942ffab` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:55.263576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84d1388-1cf9-4a6c-b02d-6cd928ec13b8 None admin] RBAC: Authorizing `identity:get_user(user_id=ac210aede44f454dbb9c3aa4a942ffab)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:55.266192 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84d1388-1cf9-4a6c-b02d-6cd928ec13b8 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:55.269414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 68/136] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:13:54 2022] GET /identity/v3/users/ac210aede44f454dbb9c3aa4a942ffab => generated 275 bytes in 441 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:55.278227 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84d1388-1cf9-4a6c-b02d-6cd928ec13b8 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:55.323181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58904907-c3f0-454c-ba4c-21c8b73a9b6a None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:55.348736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:55.366166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1b2aab-0e01-4bd2-bfba-bcb01e36b32e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:55.367229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1b2aab-0e01-4bd2-bfba-bcb01e36b32e None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:55.367776 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1b2aab-0e01-4bd2-bfba-bcb01e36b32e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:55.368254 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1b2aab-0e01-4bd2-bfba-bcb01e36b32e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:55.370047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1b2aab-0e01-4bd2-bfba-bcb01e36b32e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:55.371777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1b2aab-0e01-4bd2-bfba-bcb01e36b32e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:55.373994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 69/137] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:13:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:55.375263 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1b2aab-0e01-4bd2-bfba-bcb01e36b32e None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:55.415470 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:55.504828 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1b2aab-0e01-4bd2-bfba-bcb01e36b32e None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:55.546388 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437f4dd2-9463-483d-a1fe-a3c8a1e49065 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:55.547416 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437f4dd2-9463-483d-a1fe-a3c8a1e49065 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:55.547907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437f4dd2-9463-483d-a1fe-a3c8a1e49065 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:55.548415 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437f4dd2-9463-483d-a1fe-a3c8a1e49065 None admin] PATH_INFO: `/v3/projects/cdc2e463faa84c7fb617e11187fa0a6c` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:55.550199 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437f4dd2-9463-483d-a1fe-a3c8a1e49065 None admin] RBAC: Authorizing `identity:get_project(project_id=cdc2e463faa84c7fb617e11187fa0a6c)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:55.551940 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437f4dd2-9463-483d-a1fe-a3c8a1e49065 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:55.555128 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 70/138] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:13:55 2022] GET /identity/v3/projects/cdc2e463faa84c7fb617e11187fa0a6c => generated 312 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:55.556502 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437f4dd2-9463-483d-a1fe-a3c8a1e49065 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:55.623318 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84d1388-1cf9-4a6c-b02d-6cd928ec13b8 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:55.633042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:55.683316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437f4dd2-9463-483d-a1fe-a3c8a1e49065 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:55.713684 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72457d18-c3ea-42de-b560-2257cd13d11a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:55.714297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72457d18-c3ea-42de-b560-2257cd13d11a None admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:55.714587 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72457d18-c3ea-42de-b560-2257cd13d11a None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:55.714946 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72457d18-c3ea-42de-b560-2257cd13d11a None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85/users/4fff265355db498da98d8cb52b62f573/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:55.717062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72457d18-c3ea-42de-b560-2257cd13d11a None admin] RBAC: Authorizing `identity:create_grant(project_id=957c97404bb9448e9d03eee0d3225b85, user_id=4fff265355db498da98d8cb52b62f573, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:55.718687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72457d18-c3ea-42de-b560-2257cd13d11a None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:55.727354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a527c4-04c8-4130-aeb4-8342e7ae8e00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:55.728430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a527c4-04c8-4130-aeb4-8342e7ae8e00 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:55.728923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a527c4-04c8-4130-aeb4-8342e7ae8e00 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:55.729589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a527c4-04c8-4130-aeb4-8342e7ae8e00 None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:55.730631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a527c4-04c8-4130-aeb4-8342e7ae8e00 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:55.732385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a527c4-04c8-4130-aeb4-8342e7ae8e00 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:55.763506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 71/139] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:13:55 2022] POST /identity/v3/users => generated 283 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:13:55.764929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a527c4-04c8-4130-aeb4-8342e7ae8e00 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:55.767239 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 69/140] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:13:55 2022] PUT /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85/users/4fff265355db498da98d8cb52b62f573/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 490 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:13:55.904827 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a527c4-04c8-4130-aeb4-8342e7ae8e00 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:55.921506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:55.951243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4941f846-af44-49bf-9a80-857eaca1d13a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:55.952034 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4941f846-af44-49bf-9a80-857eaca1d13a None admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:55.952294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4941f846-af44-49bf-9a80-857eaca1d13a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:55.952660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4941f846-af44-49bf-9a80-857eaca1d13a None admin] PATH_INFO: `/v3/projects/cdc2e463faa84c7fb617e11187fa0a6c/users/ac210aede44f454dbb9c3aa4a942ffab/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:55.954850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4941f846-af44-49bf-9a80-857eaca1d13a None admin] RBAC: Authorizing `identity:create_grant(project_id=cdc2e463faa84c7fb617e11187fa0a6c, user_id=ac210aede44f454dbb9c3aa4a942ffab, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:55.956463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4941f846-af44-49bf-9a80-857eaca1d13a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:55.976593 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 72/141] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:13:55 2022] PUT /identity/v3/projects/cdc2e463faa84c7fb617e11187fa0a6c/users/ac210aede44f454dbb9c3aa4a942ffab/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:13:56.453505 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e0e917-b32d-40d9-b947-432d4a5af378 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:56.455098 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e0e917-b32d-40d9-b947-432d4a5af378 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:56.455715 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e0e917-b32d-40d9-b947-432d4a5af378 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:56.458278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 70/142] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:56 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:56.467098 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb2e427-4a03-46bd-8fe5-8104c6ad7450 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:56.467538 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb2e427-4a03-46bd-8fe5-8104c6ad7450 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:56.467841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb2e427-4a03-46bd-8fe5-8104c6ad7450 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:56.506266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-bfb2e427-4a03-46bd-8fe5-8104c6ad7450 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:56.508861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-bfb2e427-4a03-46bd-8fe5-8104c6ad7450 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:56.682310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 73/143] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:13:57.344940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e0e917-b32d-40d9-b947-432d4a5af378 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:57.421298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6088b5e1-cc0b-4d9e-8c25-7b3d11d9f49d None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:57.422081 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6088b5e1-cc0b-4d9e-8c25-7b3d11d9f49d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:57.422213 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6088b5e1-cc0b-4d9e-8c25-7b3d11d9f49d None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:57.424700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 74/144] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:57 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:13:57.432048 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860ba8a6-150a-4e46-a262-8d4d92e9e96f None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:57.432569 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860ba8a6-150a-4e46-a262-8d4d92e9e96f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:57.433057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860ba8a6-150a-4e46-a262-8d4d92e9e96f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:57.486712 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-860ba8a6-150a-4e46-a262-8d4d92e9e96f None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:57.488833 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-860ba8a6-150a-4e46-a262-8d4d92e9e96f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:57.678848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 75/145] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 249 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:13:57.680606 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:57.696604 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e0e917-b32d-40d9-b947-432d4a5af378 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:57.785856 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3766cb9f-4154-473b-b3ef-4dc52751f0d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:57.787555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3766cb9f-4154-473b-b3ef-4dc52751f0d0 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:57.788064 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3766cb9f-4154-473b-b3ef-4dc52751f0d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:57.788494 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3766cb9f-4154-473b-b3ef-4dc52751f0d0 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:57.819588 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3766cb9f-4154-473b-b3ef-4dc52751f0d0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:13:57.834119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-3766cb9f-4154-473b-b3ef-4dc52751f0d0 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:13:57.834119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:13:57.834119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:13:57.834119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:13:57.834119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:13:57.834119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:13:57.834119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:13:57.834119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:13:57.834119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:13:57.834119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:13:57.834119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:13:57.834119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:13:57.834119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:13:57.834119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:13:57.834119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:13:57.834119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:13:57.835139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:13:57.835139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:13:57.835139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:13:57.835139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:13:57.835139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:13:57.835139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:13:57.835139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:13:57.835139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:13:57.835139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:13:57.835139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:13:57.835139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:13:57.835139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:13:57.835139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:13:57.835139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:13:57.835139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:13:57.835139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:13:57.835916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:13:57.835916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:13:57.835916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:13:57.835916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:13:57.835916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:13:57.835916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:13:57.835916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:13:57.835916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:13:57.835916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:13:57.835916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:13:57.835916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:13:57.835916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:13:57.837047 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 71/146] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:13:57 2022] GET /identity/v3/roles/reader => generated 84 bytes in 493 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:13:57.842852 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860ba8a6-150a-4e46-a262-8d4d92e9e96f None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:57.899823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:57.997077 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860ba8a6-150a-4e46-a262-8d4d92e9e96f None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:58.041641 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cd30f5-b4dc-4303-b79f-481345a04ad7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:58.042272 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cd30f5-b4dc-4303-b79f-481345a04ad7 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:58.042538 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cd30f5-b4dc-4303-b79f-481345a04ad7 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:58.042811 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cd30f5-b4dc-4303-b79f-481345a04ad7 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:58.043492 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cd30f5-b4dc-4303-b79f-481345a04ad7 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:58.043789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cd30f5-b4dc-4303-b79f-481345a04ad7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:58.045255 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cd30f5-b4dc-4303-b79f-481345a04ad7 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:58.061655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 76/147] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:13:57 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:58.068866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3766cb9f-4154-473b-b3ef-4dc52751f0d0 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:58.267976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cd30f5-b4dc-4303-b79f-481345a04ad7 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:58.323434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3766cb9f-4154-473b-b3ef-4dc52751f0d0 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:58.402754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a37f1dc-f4c3-48a8-b7f9-80c535e3e511 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:58.403777 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a37f1dc-f4c3-48a8-b7f9-80c535e3e511 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:58.403777 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a37f1dc-f4c3-48a8-b7f9-80c535e3e511 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:58.404008 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a37f1dc-f4c3-48a8-b7f9-80c535e3e511 None admin] PATH_INFO: `/v3/users/229bce563b8c45fb8a24a170a209bf2f` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:58.405309 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a37f1dc-f4c3-48a8-b7f9-80c535e3e511 None admin] RBAC: Authorizing `identity:get_user(user_id=229bce563b8c45fb8a24a170a209bf2f)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:58.406972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a37f1dc-f4c3-48a8-b7f9-80c535e3e511 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:58.410417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 72/148] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:13:58 2022] GET /identity/v3/users/229bce563b8c45fb8a24a170a209bf2f => generated 289 bytes in 343 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:58.415627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a37f1dc-f4c3-48a8-b7f9-80c535e3e511 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:58.465640 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:58.533703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:58.555112 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cd30f5-b4dc-4303-b79f-481345a04ad7 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:58.624556 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a37f1dc-f4c3-48a8-b7f9-80c535e3e511 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:58.629238 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3db4ae9-2e18-4e61-8818-38ef025e9cf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:58.629907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3db4ae9-2e18-4e61-8818-38ef025e9cf2 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:58.630186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3db4ae9-2e18-4e61-8818-38ef025e9cf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:58.631027 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3db4ae9-2e18-4e61-8818-38ef025e9cf2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:58.662252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3db4ae9-2e18-4e61-8818-38ef025e9cf2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:13:58.669301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-a3db4ae9-2e18-4e61-8818-38ef025e9cf2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:13:58.669301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:13:58.669301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:13:58.669301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:13:58.669301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:13:58.669301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:13:58.669301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:13:58.669301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:13:58.669301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:13:58.669301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:13:58.669301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:13:58.669301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:13:58.669301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:13:58.669301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:13:58.669301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:13:58.669301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:13:58.672786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:13:58.672786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:13:58.672786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:13:58.672786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:13:58.672786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:13:58.672786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:13:58.672786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:13:58.672786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:13:58.672786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:13:58.672786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:13:58.672786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:13:58.672786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:13:58.672786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:13:58.672786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:13:58.672786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:13:58.672786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:13:58.675518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:13:58.675518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:13:58.675518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:13:58.675518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:13:58.675518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:13:58.675518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:13:58.675518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:13:58.675518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:13:58.675518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:13:58.675518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:13:58.675518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:13:58.675518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:13:58.675518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 77/149] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:13:58 2022] GET /identity/v3/roles/member => generated 84 bytes in 404 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:13:58.684392 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3db4ae9-2e18-4e61-8818-38ef025e9cf2 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:58.709305 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d70a1fa-8c36-4487-81cf-865cda6b11d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:58.709947 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d70a1fa-8c36-4487-81cf-865cda6b11d2 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:58.710416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d70a1fa-8c36-4487-81cf-865cda6b11d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:58.710416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d70a1fa-8c36-4487-81cf-865cda6b11d2 None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:58.711615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d70a1fa-8c36-4487-81cf-865cda6b11d2 None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:58.713162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d70a1fa-8c36-4487-81cf-865cda6b11d2 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:58.715125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 73/150] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:13:58 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:13:58.721555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d70a1fa-8c36-4487-81cf-865cda6b11d2 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:58.976028 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:58.982376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3db4ae9-2e18-4e61-8818-38ef025e9cf2 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:59.006206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d70a1fa-8c36-4487-81cf-865cda6b11d2 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:59.081409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240a0d4f-8dcf-4b25-9c5c-38e469ca9ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:59.082061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240a0d4f-8dcf-4b25-9c5c-38e469ca9ab4 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:59.082340 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240a0d4f-8dcf-4b25-9c5c-38e469ca9ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:59.082713 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240a0d4f-8dcf-4b25-9c5c-38e469ca9ab4 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:59.083305 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240a0d4f-8dcf-4b25-9c5c-38e469ca9ab4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:13:59.083646 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240a0d4f-8dcf-4b25-9c5c-38e469ca9ab4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:59.084353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8621af4c-047c-477a-9144-ef0c5b007611 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:59.085066 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8621af4c-047c-477a-9144-ef0c5b007611 None admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:59.085200 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240a0d4f-8dcf-4b25-9c5c-38e469ca9ab4 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:59.085412 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8621af4c-047c-477a-9144-ef0c5b007611 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:59.085676 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8621af4c-047c-477a-9144-ef0c5b007611 None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85/users/229bce563b8c45fb8a24a170a209bf2f/roles/f425b98e3acf4e13b0da24b21e59cfce` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:59.087770 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8621af4c-047c-477a-9144-ef0c5b007611 None admin] RBAC: Authorizing `identity:create_grant(project_id=957c97404bb9448e9d03eee0d3225b85, user_id=229bce563b8c45fb8a24a170a209bf2f, role_id=f425b98e3acf4e13b0da24b21e59cfce)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:59.089327 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8621af4c-047c-477a-9144-ef0c5b007611 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:59.100212 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 78/151] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:13:58 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 421 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:13:59.106733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240a0d4f-8dcf-4b25-9c5c-38e469ca9ab4 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:59.118992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 74/152] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:13:58 2022] PUT /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85/users/229bce563b8c45fb8a24a170a209bf2f/roles/f425b98e3acf4e13b0da24b21e59cfce => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:13:59.136271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:59.279677 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b039597d-6da4-415b-bca8-63a858e96a86 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:59.279677 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b039597d-6da4-415b-bca8-63a858e96a86 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:59.279677 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b039597d-6da4-415b-bca8-63a858e96a86 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:59.279677 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 75/153] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:13:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:13:59.293828 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755c35c3-a836-4656-84a0-3d55e355f80b None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:59.294260 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755c35c3-a836-4656-84a0-3d55e355f80b None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:59.294411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755c35c3-a836-4656-84a0-3d55e355f80b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:59.300554 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240a0d4f-8dcf-4b25-9c5c-38e469ca9ab4 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:59.334847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-755c35c3-a836-4656-84a0-3d55e355f80b None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:13:59.340352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-755c35c3-a836-4656-84a0-3d55e355f80b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:13:59.438965 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191433b2-7f59-4bd7-83f9-547ae5099adb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:59.440637 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191433b2-7f59-4bd7-83f9-547ae5099adb None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:59.441439 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191433b2-7f59-4bd7-83f9-547ae5099adb None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:59.442186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191433b2-7f59-4bd7-83f9-547ae5099adb None admin] PATH_INFO: `/v3/users/ac210aede44f454dbb9c3aa4a942ffab` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:59.448259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191433b2-7f59-4bd7-83f9-547ae5099adb None admin] RBAC: Authorizing `identity:get_user(user_id=ac210aede44f454dbb9c3aa4a942ffab)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:13:59.455256 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191433b2-7f59-4bd7-83f9-547ae5099adb None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:13:59.458954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 79/154] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:13:59 2022] GET /identity/v3/users/ac210aede44f454dbb9c3aa4a942ffab => generated 275 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:13:59.472268 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191433b2-7f59-4bd7-83f9-547ae5099adb None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:13:59.534853 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 76/155] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:13:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:13:59.694017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:13:59.738656 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191433b2-7f59-4bd7-83f9-547ae5099adb None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:13:59.817984 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef96926-74d8-428c-8dfe-5e4ec3e631d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:13:59.818937 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef96926-74d8-428c-8dfe-5e4ec3e631d8 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:13:59.819431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef96926-74d8-428c-8dfe-5e4ec3e631d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:13:59.819873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef96926-74d8-428c-8dfe-5e4ec3e631d8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:13:59.895240 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755c35c3-a836-4656-84a0-3d55e355f80b None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:00.005955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef96926-74d8-428c-8dfe-5e4ec3e631d8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:00.010958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef96926-74d8-428c-8dfe-5e4ec3e631d8 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:00.076403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:00.110855 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755c35c3-a836-4656-84a0-3d55e355f80b None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:00.194832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c644e873-542e-4222-ae49-0fc492d29fbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:00.195544 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c644e873-542e-4222-ae49-0fc492d29fbd None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:00.195790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c644e873-542e-4222-ae49-0fc492d29fbd None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:00.196069 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c644e873-542e-4222-ae49-0fc492d29fbd None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:00.213895 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c644e873-542e-4222-ae49-0fc492d29fbd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:00.218940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-c644e873-542e-4222-ae49-0fc492d29fbd None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:00.218940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:00.218940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:00.218940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:00.218940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:00.218940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:00.218940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:00.218940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:00.218940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:00.218940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:00.218940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:00.218940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:00.218940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:00.218940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:00.218940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:00.218940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:00.220858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:00.220858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:00.220858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:00.220858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:00.220858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:00.220858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:00.220858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:00.220858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:00.220858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:00.220858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:00.220858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:00.220858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:00.220858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:00.220858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:00.220858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:00.220858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:00.222950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:00.222950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:00.222950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:00.222950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:00.222950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:00.222950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:00.222950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:00.222950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:00.222950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:00.222950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:00.222950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:00.222950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:00.228776 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 77/156] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:13:59 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 334 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:00.229777 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c644e873-542e-4222-ae49-0fc492d29fbd None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:00.252547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:00.306155 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 80/157] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:13:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 835 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 14:14:00.319815 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef96926-74d8-428c-8dfe-5e4ec3e631d8 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:00.428368 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c644e873-542e-4222-ae49-0fc492d29fbd None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:00.495285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9371b50-9607-4fbb-a4ce-a31a86107b4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:00.495949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9371b50-9607-4fbb-a4ce-a31a86107b4c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:00.496231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9371b50-9607-4fbb-a4ce-a31a86107b4c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:00.496551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9371b50-9607-4fbb-a4ce-a31a86107b4c None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:00.497245 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9371b50-9607-4fbb-a4ce-a31a86107b4c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:00.497551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9371b50-9607-4fbb-a4ce-a31a86107b4c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:00.498998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9371b50-9607-4fbb-a4ce-a31a86107b4c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:00.513048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 78/158] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:14:00 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:00.525449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9371b50-9607-4fbb-a4ce-a31a86107b4c None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:00.539627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:00.627623 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef96926-74d8-428c-8dfe-5e4ec3e631d8 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:00.721697 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9371b50-9607-4fbb-a4ce-a31a86107b4c None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:00.725486 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ee9ac6-c03b-4e5f-b670-288b875e6a1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:00.726284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ee9ac6-c03b-4e5f-b670-288b875e6a1d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:00.726417 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ee9ac6-c03b-4e5f-b670-288b875e6a1d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:00.726763 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ee9ac6-c03b-4e5f-b670-288b875e6a1d None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:00.728038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ee9ac6-c03b-4e5f-b670-288b875e6a1d None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:00.730826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ee9ac6-c03b-4e5f-b670-288b875e6a1d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:00.732733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 81/159] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:00 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 415 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:00.738946 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ee9ac6-c03b-4e5f-b670-288b875e6a1d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:00.832535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9a0a0d-8742-496e-b7bf-045f8fb51781 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:00.833719 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9a0a0d-8742-496e-b7bf-045f8fb51781 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:00.834205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9a0a0d-8742-496e-b7bf-045f8fb51781 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:00.835313 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9a0a0d-8742-496e-b7bf-045f8fb51781 None admin] PATH_INFO: `/v3/users/ac210aede44f454dbb9c3aa4a942ffab` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:00.842396 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9a0a0d-8742-496e-b7bf-045f8fb51781 None admin] RBAC: Authorizing `identity:get_user(user_id=ac210aede44f454dbb9c3aa4a942ffab)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:00.845212 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9a0a0d-8742-496e-b7bf-045f8fb51781 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:00.850484 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 79/160] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:00 2022] GET /identity/v3/users/ac210aede44f454dbb9c3aa4a942ffab => generated 275 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:00.857893 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9a0a0d-8742-496e-b7bf-045f8fb51781 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:00.860814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:00.997063 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ee9ac6-c03b-4e5f-b670-288b875e6a1d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:01.083669 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:01.101220 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23f0520-e5bb-4547-8af0-042704e574a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:01.101863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23f0520-e5bb-4547-8af0-042704e574a1 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:01.102423 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23f0520-e5bb-4547-8af0-042704e574a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:01.102712 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23f0520-e5bb-4547-8af0-042704e574a1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:01.104001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23f0520-e5bb-4547-8af0-042704e574a1 None admin] RBAC: Adding query filter params (role.id=bc6aeaa495ca4f3dbe72905c744c2233, scope.project.id=957c97404bb9448e9d03eee0d3225b85, user.id=ac210aede44f454dbb9c3aa4a942ffab) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:01.104375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23f0520-e5bb-4547-8af0-042704e574a1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:01.105912 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23f0520-e5bb-4547-8af0-042704e574a1 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:01.115428 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 82/161] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:14:00 2022] GET /identity/v3/role_assignments?user.id=ac210aede44f454dbb9c3aa4a942ffab&scope.project.id=957c97404bb9448e9d03eee0d3225b85&role.id=bc6aeaa495ca4f3dbe72905c744c2233 => generated 608 bytes in 377 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:14:01.135257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9a0a0d-8742-496e-b7bf-045f8fb51781 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:01.195568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe59e249-cbd1-42e8-b908-8de47ea78422 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:01.196221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe59e249-cbd1-42e8-b908-8de47ea78422 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:01.196533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe59e249-cbd1-42e8-b908-8de47ea78422 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:01.196803 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe59e249-cbd1-42e8-b908-8de47ea78422 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:01.295934 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe59e249-cbd1-42e8-b908-8de47ea78422 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:01.297468 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe59e249-cbd1-42e8-b908-8de47ea78422 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:01.416950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:01.453752 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 80/162] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 598 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 14:14:01.460652 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23f0520-e5bb-4547-8af0-042704e574a1 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:01.691450 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23f0520-e5bb-4547-8af0-042704e574a1 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:01.733986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:01.780834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5741b00d-ff88-45fa-a37c-5e526233b7b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:01.781979 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5741b00d-ff88-45fa-a37c-5e526233b7b3 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:01.782463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5741b00d-ff88-45fa-a37c-5e526233b7b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:01.783215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5741b00d-ff88-45fa-a37c-5e526233b7b3 None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:01.785486 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5741b00d-ff88-45fa-a37c-5e526233b7b3 None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:01.789102 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5741b00d-ff88-45fa-a37c-5e526233b7b3 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:01.796906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 83/163] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:01 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:01.808422 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe59e249-cbd1-42e8-b908-8de47ea78422 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:01.839722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5681f72e-3995-4c6e-bf81-479aa4f2fdc9 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:01.840193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5681f72e-3995-4c6e-bf81-479aa4f2fdc9 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:01.840695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5681f72e-3995-4c6e-bf81-479aa4f2fdc9 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:01.844710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 84/164] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:01 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:01.862686 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec934b56-f7eb-419e-a6a9-b36960dd1cb5 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:01.863131 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec934b56-f7eb-419e-a6a9-b36960dd1cb5 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:01.863808 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec934b56-f7eb-419e-a6a9-b36960dd1cb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:01.915232 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-ec934b56-f7eb-419e-a6a9-b36960dd1cb5 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:01.917454 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ec934b56-f7eb-419e-a6a9-b36960dd1cb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:02.026308 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe59e249-cbd1-42e8-b908-8de47ea78422 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:02.089788 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a886eaa5-16e8-44d2-a3bc-77c725c1214c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:02.090866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a886eaa5-16e8-44d2-a3bc-77c725c1214c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:02.091358 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a886eaa5-16e8-44d2-a3bc-77c725c1214c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:02.091837 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a886eaa5-16e8-44d2-a3bc-77c725c1214c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:02.093237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a886eaa5-16e8-44d2-a3bc-77c725c1214c None admin] RBAC: Adding query filter params (role.id=e114d28e2dd147fcb39452dd461791c0, scope.project.id=957c97404bb9448e9d03eee0d3225b85, user.id=ac210aede44f454dbb9c3aa4a942ffab) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:02.093788 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a886eaa5-16e8-44d2-a3bc-77c725c1214c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:02.096504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a886eaa5-16e8-44d2-a3bc-77c725c1214c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:02.125489 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 81/165] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:14:01 2022] GET /identity/v3/role_assignments?user.id=ac210aede44f454dbb9c3aa4a942ffab&scope.project.id=957c97404bb9448e9d03eee0d3225b85&role.id=e114d28e2dd147fcb39452dd461791c0 => generated 608 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:02.156216 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 85/166] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 295 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:14:02.858312 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a886eaa5-16e8-44d2-a3bc-77c725c1214c None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:02.873497 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98168f6-cf0e-4138-9e4b-02f909ed8193 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:02.874044 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98168f6-cf0e-4138-9e4b-02f909ed8193 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:02.874044 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98168f6-cf0e-4138-9e4b-02f909ed8193 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:02.877328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 86/167] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:02.883813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da717b9-cb8e-4da7-8dce-d7c2c702d3ee None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:02.884175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da717b9-cb8e-4da7-8dce-d7c2c702d3ee None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:02.884414 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da717b9-cb8e-4da7-8dce-d7c2c702d3ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:02.921279 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-1da717b9-cb8e-4da7-8dce-d7c2c702d3ee None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:02.922517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1da717b9-cb8e-4da7-8dce-d7c2c702d3ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:02.956764 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:03.092189 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a886eaa5-16e8-44d2-a3bc-77c725c1214c None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:03.114453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 87/168] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:14:03.114453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bae0faf-63b9-4af5-9221-a59a52e31064 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:03.116370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bae0faf-63b9-4af5-9221-a59a52e31064 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:03.116563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bae0faf-63b9-4af5-9221-a59a52e31064 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:03.119266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 88/169] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:03 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:03.128171 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887b4c2a-c8a9-4d0f-ad42-1cf5b829daf0 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:03.128670 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887b4c2a-c8a9-4d0f-ad42-1cf5b829daf0 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:03.129125 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887b4c2a-c8a9-4d0f-ad42-1cf5b829daf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:03.153779 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa966d4-aca3-4004-b0fc-f8777fc0da91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:03.154857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa966d4-aca3-4004-b0fc-f8777fc0da91 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:03.155345 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa966d4-aca3-4004-b0fc-f8777fc0da91 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:03.155819 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa966d4-aca3-4004-b0fc-f8777fc0da91 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:03.169638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa966d4-aca3-4004-b0fc-f8777fc0da91 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:03.178518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-1aa966d4-aca3-4004-b0fc-f8777fc0da91 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:14:03.178518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:03.178518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:03.178518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:03.178518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:03.178518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:03.178518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:03.178518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:03.178518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:03.178518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:03.178518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:03.178518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:03.178518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:03.178518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:03.178518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:03.178518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:03.179485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:03.179485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:03.179485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:03.179485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:03.179485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:03.179485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:03.179485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:03.179485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:03.179485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:03.179485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:03.179485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:03.179485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:03.179485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:03.179485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:03.179485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:03.179485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:03.180289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:03.180289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:03.180289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:03.180289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:03.180289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:03.180289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:03.180289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:03.180289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:03.180289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:03.180289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:03.180289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:14:03.180289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:03.180933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-887b4c2a-c8a9-4d0f-ad42-1cf5b829daf0 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:03.182861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 82/170] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:14:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 325 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:03.187429 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-887b4c2a-c8a9-4d0f-ad42-1cf5b829daf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:03.189833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa966d4-aca3-4004-b0fc-f8777fc0da91 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:03.261961 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:03.317132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:03.323400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa966d4-aca3-4004-b0fc-f8777fc0da91 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:03.335783 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 89/171] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:14:03.367292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e89e47-5540-4db2-ae7c-e3cff9667ef5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:03.368417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e89e47-5540-4db2-ae7c-e3cff9667ef5 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:03.368910 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e89e47-5540-4db2-ae7c-e3cff9667ef5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:03.369379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e89e47-5540-4db2-ae7c-e3cff9667ef5 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:03.370341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e89e47-5540-4db2-ae7c-e3cff9667ef5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:03.370862 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e89e47-5540-4db2-ae7c-e3cff9667ef5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:03.372557 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e89e47-5540-4db2-ae7c-e3cff9667ef5 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:03.383759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 83/172] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:14:03.391490 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887b4c2a-c8a9-4d0f-ad42-1cf5b829daf0 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:03.548968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887b4c2a-c8a9-4d0f-ad42-1cf5b829daf0 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:03.591226 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43043816-2972-4a58-9b74-62c52c2b03dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:03.591912 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43043816-2972-4a58-9b74-62c52c2b03dc None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:03.592200 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43043816-2972-4a58-9b74-62c52c2b03dc None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:03.592563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43043816-2972-4a58-9b74-62c52c2b03dc None admin] PATH_INFO: `/v3/users/4fff265355db498da98d8cb52b62f573` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:03.593982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43043816-2972-4a58-9b74-62c52c2b03dc None admin] RBAC: Authorizing `identity:get_user(user_id=4fff265355db498da98d8cb52b62f573)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:03.595504 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43043816-2972-4a58-9b74-62c52c2b03dc None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:03.597503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 90/173] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:03 2022] GET /identity/v3/users/4fff265355db498da98d8cb52b62f573 => generated 247 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:03.603861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e89e47-5540-4db2-ae7c-e3cff9667ef5 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:03.736195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d25e3c-5d2d-4cfd-b273-2345f4df0a69 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:03.736526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d25e3c-5d2d-4cfd-b273-2345f4df0a69 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:03.736767 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d25e3c-5d2d-4cfd-b273-2345f4df0a69 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:03.738161 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 91/174] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:03.754198 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb600aa5-b3fd-47f4-9948-140bb65fd410 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:03.754198 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb600aa5-b3fd-47f4-9948-140bb65fd410 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:03.754456 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb600aa5-b3fd-47f4-9948-140bb65fd410 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:03.761967 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e89e47-5540-4db2-ae7c-e3cff9667ef5 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:03.797740 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-eb600aa5-b3fd-47f4-9948-140bb65fd410 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:03.799009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-eb600aa5-b3fd-47f4-9948-140bb65fd410 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:03.806464 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c92d2ef-95b9-4c32-b3c0-fa8c65c8ffa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:03.807449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c92d2ef-95b9-4c32-b3c0-fa8c65c8ffa5 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:03.807987 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c92d2ef-95b9-4c32-b3c0-fa8c65c8ffa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:03.808501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c92d2ef-95b9-4c32-b3c0-fa8c65c8ffa5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:03.843004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:03.845038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:03.900348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c92d2ef-95b9-4c32-b3c0-fa8c65c8ffa5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:03.900348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c92d2ef-95b9-4c32-b3c0-fa8c65c8ffa5 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:03.917963 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 92/175] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:03.919628 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb600aa5-b3fd-47f4-9948-140bb65fd410 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:04.046011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb600aa5-b3fd-47f4-9948-140bb65fd410 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:04.061518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 84/176] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 458 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 14:14:04.069665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c92d2ef-95b9-4c32-b3c0-fa8c65c8ffa5 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:04.093480 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85628d59-c912-488f-8b68-862f2493ff81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:04.094834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85628d59-c912-488f-8b68-862f2493ff81 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:04.095392 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85628d59-c912-488f-8b68-862f2493ff81 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:04.095868 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85628d59-c912-488f-8b68-862f2493ff81 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:04.110243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85628d59-c912-488f-8b68-862f2493ff81 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:04.116914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-85628d59-c912-488f-8b68-862f2493ff81 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:14:04.116914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:04.116914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:04.116914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:04.116914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:04.116914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:04.116914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:04.116914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:04.116914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:04.116914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:04.116914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:04.116914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:04.116914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:04.116914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:04.116914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:04.116914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:04.117941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:04.117941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:04.117941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:04.117941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:04.117941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:04.117941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:04.117941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:04.117941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:04.117941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:04.117941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:04.117941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:04.117941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:04.117941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:04.117941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:04.117941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:04.117941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:04.118874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:04.118874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:04.118874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:04.118874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:04.118874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:04.118874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:04.118874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:04.118874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:04.118874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:04.118874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:04.118874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:14:04.118874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:04.121475 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 93/177] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:14:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:04.122744 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85628d59-c912-488f-8b68-862f2493ff81 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:04.163761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:04.196113 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:04.256185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85628d59-c912-488f-8b68-862f2493ff81 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:04.272819 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c92d2ef-95b9-4c32-b3c0-fa8c65c8ffa5 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:04.300107 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0790ff1b-2f43-46e8-9240-82d415865422 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:04.301162 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0790ff1b-2f43-46e8-9240-82d415865422 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:04.301447 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0790ff1b-2f43-46e8-9240-82d415865422 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:04.302264 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0790ff1b-2f43-46e8-9240-82d415865422 None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:04.303680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0790ff1b-2f43-46e8-9240-82d415865422 None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:04.305198 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0790ff1b-2f43-46e8-9240-82d415865422 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:04.307129 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 94/178] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:04 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:04.307852 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0790ff1b-2f43-46e8-9240-82d415865422 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:04.319186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205df74d-e7f1-47cc-90d1-f2e9fa881146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:04.319852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205df74d-e7f1-47cc-90d1-f2e9fa881146 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:04.320139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205df74d-e7f1-47cc-90d1-f2e9fa881146 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:04.320445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205df74d-e7f1-47cc-90d1-f2e9fa881146 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:04.331998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-205df74d-e7f1-47cc-90d1-f2e9fa881146 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:04.337753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-205df74d-e7f1-47cc-90d1-f2e9fa881146 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:04.337753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:04.337753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:04.337753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:04.337753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:04.337753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:04.337753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:04.337753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:04.337753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:04.337753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:04.337753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:04.337753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:04.337753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:04.337753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:04.337753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:04.337753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:04.338756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:04.338756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:04.338756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:04.338756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:04.338756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:04.338756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:04.338756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:04.338756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:04.338756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:04.338756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:04.338756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:04.338756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:04.338756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:04.338756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:04.338756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:04.338756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:04.339555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:04.339555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:04.339555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:04.339555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:04.339555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:04.339555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:04.339555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:04.339555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:04.339555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:04.339555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:04.339555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:04.339555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:04.339555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 85/179] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:04 2022] GET /identity/v3/roles/member => generated 84 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:04.340211 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205df74d-e7f1-47cc-90d1-f2e9fa881146 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:04.436965 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:04.444893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0790ff1b-2f43-46e8-9240-82d415865422 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:04.474820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205df74d-e7f1-47cc-90d1-f2e9fa881146 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:04.491125 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf916a5-5eeb-4d5e-8b07-1942c1f06952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:04.491749 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf916a5-5eeb-4d5e-8b07-1942c1f06952 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:04.492026 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf916a5-5eeb-4d5e-8b07-1942c1f06952 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:04.492299 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf916a5-5eeb-4d5e-8b07-1942c1f06952 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:04.493023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf916a5-5eeb-4d5e-8b07-1942c1f06952 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:04.493327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf916a5-5eeb-4d5e-8b07-1942c1f06952 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:04.494780 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf916a5-5eeb-4d5e-8b07-1942c1f06952 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:04.507654 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 95/180] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:04.509305 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41a11a7-73b2-4fe4-b62c-048e7f4b1af3 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:04.509600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41a11a7-73b2-4fe4-b62c-048e7f4b1af3 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:04.509849 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41a11a7-73b2-4fe4-b62c-048e7f4b1af3 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:04.511268 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 96/181] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:04.515538 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41a11a7-73b2-4fe4-b62c-048e7f4b1af3 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:04.519821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:04.541225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be3be66-da12-42d1-9921-28109c2d5944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:04.541874 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be3be66-da12-42d1-9921-28109c2d5944 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:04.542154 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be3be66-da12-42d1-9921-28109c2d5944 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:04.542428 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be3be66-da12-42d1-9921-28109c2d5944 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:04.555275 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be3be66-da12-42d1-9921-28109c2d5944 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:04.560989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-7be3be66-da12-42d1-9921-28109c2d5944 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:04.560989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:04.560989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:04.560989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:04.560989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:04.560989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:04.560989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:04.560989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:04.560989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:04.560989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:04.560989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:04.560989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:04.560989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:04.560989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:04.560989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:04.560989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:04.564873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:04.564873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:04.564873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:04.564873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:04.564873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:04.564873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:04.564873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:04.564873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:04.564873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:04.564873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:04.564873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:04.564873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:04.564873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:04.564873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:04.564873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:04.564873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:04.565714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:04.565714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:04.565714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:04.565714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:04.565714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:04.565714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:04.565714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:04.565714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:04.565714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:04.565714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:04.565714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:04.565714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:04.565714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 86/182] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:14:04 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:04.565714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be3be66-da12-42d1-9921-28109c2d5944 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:04.687916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41a11a7-73b2-4fe4-b62c-048e7f4b1af3 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:04.741916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dab6c7-c679-4d36-a72a-c2bd88b2d11f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:04.742540 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dab6c7-c679-4d36-a72a-c2bd88b2d11f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:04.742819 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dab6c7-c679-4d36-a72a-c2bd88b2d11f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:04.743089 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dab6c7-c679-4d36-a72a-c2bd88b2d11f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:04.743831 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10dab6c7-c679-4d36-a72a-c2bd88b2d11f None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=957c97404bb9448e9d03eee0d3225b85, user.id=4fff265355db498da98d8cb52b62f573) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:04.744139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10dab6c7-c679-4d36-a72a-c2bd88b2d11f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:04.745732 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10dab6c7-c679-4d36-a72a-c2bd88b2d11f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:04.764423 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 97/183] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:14:04 2022] GET /identity/v3/role_assignments?user.id=4fff265355db498da98d8cb52b62f573&scope.project.id=957c97404bb9448e9d03eee0d3225b85&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 608 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:04.764423 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dab6c7-c679-4d36-a72a-c2bd88b2d11f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:04.764841 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be3be66-da12-42d1-9921-28109c2d5944 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:04.790106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:04.814433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd2b8af-889c-4414-b222-09b30b7cfe94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:04.815330 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd2b8af-889c-4414-b222-09b30b7cfe94 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:04.815804 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd2b8af-889c-4414-b222-09b30b7cfe94 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:04.816269 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd2b8af-889c-4414-b222-09b30b7cfe94 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:04.817520 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd2b8af-889c-4414-b222-09b30b7cfe94 None admin] RBAC: Adding query filter params (name=member) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:04.818024 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd2b8af-889c-4414-b222-09b30b7cfe94 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:04.820234 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd2b8af-889c-4414-b222-09b30b7cfe94 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:04.836375 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 87/184] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:04 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:04.837369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeedc63-7a8e-41f5-9726-255540aa238f None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:04.837855 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeedc63-7a8e-41f5-9726-255540aa238f None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:04.838341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeedc63-7a8e-41f5-9726-255540aa238f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:04.865811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-edeedc63-7a8e-41f5-9726-255540aa238f None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:04.867375 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-edeedc63-7a8e-41f5-9726-255540aa238f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:04.916937 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:04.930185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dab6c7-c679-4d36-a72a-c2bd88b2d11f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:04.976573 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b10ca7-66d9-42b9-94d3-a1990ec9f4a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:04.977223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b10ca7-66d9-42b9-94d3-a1990ec9f4a4 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:04.977501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b10ca7-66d9-42b9-94d3-a1990ec9f4a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:04.977776 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b10ca7-66d9-42b9-94d3-a1990ec9f4a4 None admin] PATH_INFO: `/v3/users/21ad892cf1a54f64933e32be81fa2dc1` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:04.984088 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 88/185] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:04.985533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edeedc63-7a8e-41f5-9726-255540aa238f None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:05.004343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b10ca7-66d9-42b9-94d3-a1990ec9f4a4 None admin] RBAC: Authorizing `identity:get_user(user_id=21ad892cf1a54f64933e32be81fa2dc1)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:05.005526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b10ca7-66d9-42b9-94d3-a1990ec9f4a4 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:05.010117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 98/186] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:04 2022] GET /identity/v3/users/21ad892cf1a54f64933e32be81fa2dc1 => generated 283 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:05.017939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afdd76c-77aa-4d5a-b9f8-03dc373bc646 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:05.018223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afdd76c-77aa-4d5a-b9f8-03dc373bc646 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:05.018497 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afdd76c-77aa-4d5a-b9f8-03dc373bc646 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:05.019923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 99/187] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:05.020753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afdd76c-77aa-4d5a-b9f8-03dc373bc646 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:05.171487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:05.195945 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edeedc63-7a8e-41f5-9726-255540aa238f None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:05.196147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afdd76c-77aa-4d5a-b9f8-03dc373bc646 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:05.243130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1144fc4-f074-415b-bcf5-a769d1bd6ad7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:05.243912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1144fc4-f074-415b-bcf5-a769d1bd6ad7 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:05.244197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1144fc4-f074-415b-bcf5-a769d1bd6ad7 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:05.244528 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1144fc4-f074-415b-bcf5-a769d1bd6ad7 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:05.245199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1144fc4-f074-415b-bcf5-a769d1bd6ad7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:05.245502 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1144fc4-f074-415b-bcf5-a769d1bd6ad7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:05.246929 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1144fc4-f074-415b-bcf5-a769d1bd6ad7 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:05.249501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883be87c-cb22-41be-a5cc-33ca12738648 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:05.250521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883be87c-cb22-41be-a5cc-33ca12738648 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:05.250999 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883be87c-cb22-41be-a5cc-33ca12738648 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:05.251474 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883be87c-cb22-41be-a5cc-33ca12738648 None admin] PATH_INFO: `/v3/users/ac210aede44f454dbb9c3aa4a942ffab` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:05.253473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883be87c-cb22-41be-a5cc-33ca12738648 None admin] RBAC: Authorizing `identity:get_user(user_id=ac210aede44f454dbb9c3aa4a942ffab)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:05.261955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883be87c-cb22-41be-a5cc-33ca12738648 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:05.262140 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 89/188] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:14:04 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:05.262140 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1144fc4-f074-415b-bcf5-a769d1bd6ad7 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:05.262273 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 100/189] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:05 2022] GET /identity/v3/users/ac210aede44f454dbb9c3aa4a942ffab => generated 275 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:05.262273 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429a14e4-2efb-4ca1-9add-86da39ba3b54 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:05.262273 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429a14e4-2efb-4ca1-9add-86da39ba3b54 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:05.262826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429a14e4-2efb-4ca1-9add-86da39ba3b54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:05.298020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-429a14e4-2efb-4ca1-9add-86da39ba3b54 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:05.299984 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-429a14e4-2efb-4ca1-9add-86da39ba3b54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:05.319014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:05.415151 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 101/190] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:05.416637 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429a14e4-2efb-4ca1-9add-86da39ba3b54 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:05.434153 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1144fc4-f074-415b-bcf5-a769d1bd6ad7 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:05.490289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d4e30e-565a-4c4a-bcd8-6c3c9b13c3a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:05.491082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d4e30e-565a-4c4a-bcd8-6c3c9b13c3a4 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:05.491435 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d4e30e-565a-4c4a-bcd8-6c3c9b13c3a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:05.491778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d4e30e-565a-4c4a-bcd8-6c3c9b13c3a4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:05.557611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:05.585112 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429a14e4-2efb-4ca1-9add-86da39ba3b54 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:05.607706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d4e30e-565a-4c4a-bcd8-6c3c9b13c3a4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:05.609251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d4e30e-565a-4c4a-bcd8-6c3c9b13c3a4 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:05.636284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53f2b47-12a7-449a-ab23-9afeca56f35c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:05.637372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53f2b47-12a7-449a-ab23-9afeca56f35c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:05.637877 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53f2b47-12a7-449a-ab23-9afeca56f35c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:05.638446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53f2b47-12a7-449a-ab23-9afeca56f35c None admin] PATH_INFO: `/v3/users/21ad892cf1a54f64933e32be81fa2dc1` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:05.640196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53f2b47-12a7-449a-ab23-9afeca56f35c None admin] RBAC: Authorizing `identity:get_user(user_id=21ad892cf1a54f64933e32be81fa2dc1)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:05.641851 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53f2b47-12a7-449a-ab23-9afeca56f35c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:05.649870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 102/191] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:05 2022] GET /identity/v3/users/21ad892cf1a54f64933e32be81fa2dc1 => generated 283 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:05.650939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53f2b47-12a7-449a-ab23-9afeca56f35c None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:05.656854 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:05.758083 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 90/192] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 500 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:14:05.759476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d4e30e-565a-4c4a-bcd8-6c3c9b13c3a4 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:05.822086 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53f2b47-12a7-449a-ab23-9afeca56f35c None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:05.890776 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2b1f3a-a6da-406a-a47b-b3196f96431b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:05.891739 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2b1f3a-a6da-406a-a47b-b3196f96431b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:05.892195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2b1f3a-a6da-406a-a47b-b3196f96431b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:05.892689 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2b1f3a-a6da-406a-a47b-b3196f96431b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:05.925598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:05.943018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d4e30e-565a-4c4a-bcd8-6c3c9b13c3a4 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:06.000616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedcf214-6cb0-4f3e-86ce-cd73f6fe48d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:06.001624 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedcf214-6cb0-4f3e-86ce-cd73f6fe48d6 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:06.002083 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedcf214-6cb0-4f3e-86ce-cd73f6fe48d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:06.002464 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedcf214-6cb0-4f3e-86ce-cd73f6fe48d6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:06.004231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedcf214-6cb0-4f3e-86ce-cd73f6fe48d6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:06.007775 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedcf214-6cb0-4f3e-86ce-cd73f6fe48d6 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:06.011037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 91/193] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:14:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:06.013585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedcf214-6cb0-4f3e-86ce-cd73f6fe48d6 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:06.016888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2b1f3a-a6da-406a-a47b-b3196f96431b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:06.017629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:06.019213 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2b1f3a-a6da-406a-a47b-b3196f96431b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:06.158064 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 103/194] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 508 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:14:06.158343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedcf214-6cb0-4f3e-86ce-cd73f6fe48d6 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:06.159203 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2b1f3a-a6da-406a-a47b-b3196f96431b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:06.226883 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2536883e-1d03-427f-9b3c-ffd2f65ec50d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:06.227971 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2536883e-1d03-427f-9b3c-ffd2f65ec50d None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:06.228516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2536883e-1d03-427f-9b3c-ffd2f65ec50d None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:06.229005 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2536883e-1d03-427f-9b3c-ffd2f65ec50d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:06.232078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:06.311651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2b1f3a-a6da-406a-a47b-b3196f96431b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:06.322458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2536883e-1d03-427f-9b3c-ffd2f65ec50d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:06.324344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2536883e-1d03-427f-9b3c-ffd2f65ec50d None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:06.327051 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:06.355875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0404dc12-e9a7-47e1-8f54-7cc305a334d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:06.356546 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0404dc12-e9a7-47e1-8f54-7cc305a334d2 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:06.356833 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0404dc12-e9a7-47e1-8f54-7cc305a334d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:06.357092 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0404dc12-e9a7-47e1-8f54-7cc305a334d2 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:06.358262 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0404dc12-e9a7-47e1-8f54-7cc305a334d2 None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:06.359748 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0404dc12-e9a7-47e1-8f54-7cc305a334d2 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:06.361521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 104/195] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:06 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:06.362331 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0404dc12-e9a7-47e1-8f54-7cc305a334d2 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:06.468114 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 92/196] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:14:06.469095 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2536883e-1d03-427f-9b3c-ffd2f65ec50d None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:06.503478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0404dc12-e9a7-47e1-8f54-7cc305a334d2 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:06.535600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:06.565754 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd02f929-dee8-4553-87a7-a5ee59c3d690 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:06.566295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd02f929-dee8-4553-87a7-a5ee59c3d690 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:06.566562 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd02f929-dee8-4553-87a7-a5ee59c3d690 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:06.566834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd02f929-dee8-4553-87a7-a5ee59c3d690 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:06.580042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd02f929-dee8-4553-87a7-a5ee59c3d690 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:06.587002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-bd02f929-dee8-4553-87a7-a5ee59c3d690 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:14:06.587002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:06.587002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:06.587002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:06.587002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:06.587002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:06.587002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:06.587002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:06.587002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:06.587002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:06.587002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:06.587002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:06.587002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:06.587002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:06.587002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:06.587002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:06.588722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:06.588722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:06.588722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:06.588722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:06.588722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:06.588722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:06.588722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:06.588722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:06.588722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:06.588722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:06.588722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:06.588722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:06.588722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:06.588722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:06.588722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:06.588722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:06.590591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:06.590591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:06.590591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:06.590591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:06.590591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:06.590591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:06.590591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:06.590591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:06.590591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:06.590591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:06.590591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:14:06.590591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:06.590591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 105/197] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:06 2022] GET /identity/v3/roles/reader => generated 84 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:06.590591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd02f929-dee8-4553-87a7-a5ee59c3d690 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:06.610868 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2536883e-1d03-427f-9b3c-ffd2f65ec50d None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:06.612826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:06.669695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29a6bf9-7719-48d1-8936-5beb34e3d7ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:06.670512 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29a6bf9-7719-48d1-8936-5beb34e3d7ac None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:06.670881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29a6bf9-7719-48d1-8936-5beb34e3d7ac None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:06.671216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29a6bf9-7719-48d1-8936-5beb34e3d7ac None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:06.671961 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29a6bf9-7719-48d1-8936-5beb34e3d7ac None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:06.673620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29a6bf9-7719-48d1-8936-5beb34e3d7ac None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:06.705439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 93/198] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:14:06 2022] POST /identity/v3/users => generated 297 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:14:06.706642 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29a6bf9-7719-48d1-8936-5beb34e3d7ac None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:06.747902 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd02f929-dee8-4553-87a7-a5ee59c3d690 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:06.799430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf08751-2bb7-4f25-a82b-a17bd2cff310 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:06.800478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf08751-2bb7-4f25-a82b-a17bd2cff310 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:06.800870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf08751-2bb7-4f25-a82b-a17bd2cff310 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:06.801326 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf08751-2bb7-4f25-a82b-a17bd2cff310 None admin] PATH_INFO: `/v3/projects/cdc2e463faa84c7fb617e11187fa0a6c` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:06.803670 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf08751-2bb7-4f25-a82b-a17bd2cff310 None admin] RBAC: Authorizing `identity:get_project(project_id=cdc2e463faa84c7fb617e11187fa0a6c)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:06.805543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf08751-2bb7-4f25-a82b-a17bd2cff310 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:06.815139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 106/199] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:06 2022] GET /identity/v3/projects/cdc2e463faa84c7fb617e11187fa0a6c => generated 312 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:06.815139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf08751-2bb7-4f25-a82b-a17bd2cff310 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:06.882312 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:06.922272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29a6bf9-7719-48d1-8936-5beb34e3d7ac None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:06.975681 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf08751-2bb7-4f25-a82b-a17bd2cff310 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:06.988630 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1f1bec-a7da-4e80-9457-3cd8953c06bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:06.990111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1f1bec-a7da-4e80-9457-3cd8953c06bf None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:06.990597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1f1bec-a7da-4e80-9457-3cd8953c06bf None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:06.990890 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1f1bec-a7da-4e80-9457-3cd8953c06bf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:06.991678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1f1bec-a7da-4e80-9457-3cd8953c06bf None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535, user.id=21ad892cf1a54f64933e32be81fa2dc1) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:06.991988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1f1bec-a7da-4e80-9457-3cd8953c06bf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:06.993537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1f1bec-a7da-4e80-9457-3cd8953c06bf None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:07.002109 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 94/200] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:14:06 2022] GET /identity/v3/role_assignments?user.id=21ad892cf1a54f64933e32be81fa2dc1&scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 259 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:07.006532 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1f1bec-a7da-4e80-9457-3cd8953c06bf None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:07.033563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e666b0d2-a1d0-47fd-925a-138e199778c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:07.034349 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e666b0d2-a1d0-47fd-925a-138e199778c3 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:07.034522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e666b0d2-a1d0-47fd-925a-138e199778c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:07.034880 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e666b0d2-a1d0-47fd-925a-138e199778c3 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:07.036973 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e666b0d2-a1d0-47fd-925a-138e199778c3 None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:07.039109 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e666b0d2-a1d0-47fd-925a-138e199778c3 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:07.041338 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 107/201] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:06 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:07.044381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e666b0d2-a1d0-47fd-925a-138e199778c3 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:07.074567 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:07.215419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1f1bec-a7da-4e80-9457-3cd8953c06bf None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:07.269557 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b00ff6b-ca55-4e32-a483-fdae2f55f9ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:07.270195 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b00ff6b-ca55-4e32-a483-fdae2f55f9ad None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:07.271102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b00ff6b-ca55-4e32-a483-fdae2f55f9ad None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:07.271516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b00ff6b-ca55-4e32-a483-fdae2f55f9ad None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:07.272231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b00ff6b-ca55-4e32-a483-fdae2f55f9ad None admin] RBAC: Adding query filter params (name=reader) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:07.273421 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b00ff6b-ca55-4e32-a483-fdae2f55f9ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:07.275878 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b00ff6b-ca55-4e32-a483-fdae2f55f9ad None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:07.277436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e666b0d2-a1d0-47fd-925a-138e199778c3 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:07.286040 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 95/202] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:07 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:07.287913 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b00ff6b-ca55-4e32-a483-fdae2f55f9ad None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:07.329191 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:07.358723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3052276-25ef-498e-9efc-cf21d4fb1a8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:07.358723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3052276-25ef-498e-9efc-cf21d4fb1a8a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:07.359138 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3052276-25ef-498e-9efc-cf21d4fb1a8a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:07.359673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3052276-25ef-498e-9efc-cf21d4fb1a8a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:07.361510 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3052276-25ef-498e-9efc-cf21d4fb1a8a None admin] RBAC: Adding query filter params (role.id=bc6aeaa495ca4f3dbe72905c744c2233, scope.project.id=cdc2e463faa84c7fb617e11187fa0a6c, user.id=ac210aede44f454dbb9c3aa4a942ffab) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:07.362248 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3052276-25ef-498e-9efc-cf21d4fb1a8a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:07.365124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3052276-25ef-498e-9efc-cf21d4fb1a8a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:07.377536 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 108/203] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:14:07 2022] GET /identity/v3/role_assignments?user.id=ac210aede44f454dbb9c3aa4a942ffab&scope.project.id=cdc2e463faa84c7fb617e11187fa0a6c&role.id=bc6aeaa495ca4f3dbe72905c744c2233 => generated 608 bytes in 336 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:07.379012 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3052276-25ef-498e-9efc-cf21d4fb1a8a None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:07.468980 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:07.523598 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b00ff6b-ca55-4e32-a483-fdae2f55f9ad None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:07.634063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af83c4cc-23c6-410a-827a-b6ef793c3a64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:07.634889 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af83c4cc-23c6-410a-827a-b6ef793c3a64 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:07.635035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af83c4cc-23c6-410a-827a-b6ef793c3a64 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:07.635298 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af83c4cc-23c6-410a-827a-b6ef793c3a64 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:07.636097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af83c4cc-23c6-410a-827a-b6ef793c3a64 None admin] RBAC: Adding query filter params (role.id=e114d28e2dd147fcb39452dd461791c0, scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535, user.id=21ad892cf1a54f64933e32be81fa2dc1) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:07.636511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af83c4cc-23c6-410a-827a-b6ef793c3a64 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:07.638125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af83c4cc-23c6-410a-827a-b6ef793c3a64 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:07.653406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 96/204] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:14:07 2022] GET /identity/v3/role_assignments?user.id=21ad892cf1a54f64933e32be81fa2dc1&scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535&role.id=e114d28e2dd147fcb39452dd461791c0 => generated 259 bytes in 367 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:07.690823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3052276-25ef-498e-9efc-cf21d4fb1a8a None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:07.779433 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5b4663-5faf-4d37-aae4-10a58a2e9f80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:07.781110 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5b4663-5faf-4d37-aae4-10a58a2e9f80 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:07.781898 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5b4663-5faf-4d37-aae4-10a58a2e9f80 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:07.782926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5b4663-5faf-4d37-aae4-10a58a2e9f80 None admin] PATH_INFO: `/v3/users/229bce563b8c45fb8a24a170a209bf2f` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:07.785655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5b4663-5faf-4d37-aae4-10a58a2e9f80 None admin] RBAC: Authorizing `identity:get_user(user_id=229bce563b8c45fb8a24a170a209bf2f)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:07.789114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5b4663-5faf-4d37-aae4-10a58a2e9f80 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:07.792922 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 109/205] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:07 2022] GET /identity/v3/users/229bce563b8c45fb8a24a170a209bf2f => generated 289 bytes in 415 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:07.803131 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af83c4cc-23c6-410a-827a-b6ef793c3a64 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:07.980084 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af83c4cc-23c6-410a-827a-b6ef793c3a64 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:08.065806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e83651-25de-4712-be03-0a047fddcffd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:08.069149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e83651-25de-4712-be03-0a047fddcffd None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:08.070042 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e83651-25de-4712-be03-0a047fddcffd None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:08.070830 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e83651-25de-4712-be03-0a047fddcffd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:08.117159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:08.227845 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e83651-25de-4712-be03-0a047fddcffd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:08.231310 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e83651-25de-4712-be03-0a047fddcffd None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:08.488387 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 97/206] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 686 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 14:14:08.496035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5b4663-5faf-4d37-aae4-10a58a2e9f80 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:08.611529 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:08.741411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5b4663-5faf-4d37-aae4-10a58a2e9f80 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:08.825130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0b8ef8-68bf-41ee-8b35-e332a1fae2da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:08.825798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0b8ef8-68bf-41ee-8b35-e332a1fae2da None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:08.826079 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0b8ef8-68bf-41ee-8b35-e332a1fae2da None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:08.826350 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0b8ef8-68bf-41ee-8b35-e332a1fae2da None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:08.827624 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0b8ef8-68bf-41ee-8b35-e332a1fae2da None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:08.829164 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0b8ef8-68bf-41ee-8b35-e332a1fae2da None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:08.831233 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 110/207] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:08 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 336 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:08.841871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e83651-25de-4712-be03-0a047fddcffd None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:08.947110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:09.072229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e83651-25de-4712-be03-0a047fddcffd None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:09.168827 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fd2f41-ef8d-4569-9295-b410b3057cc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:09.169891 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fd2f41-ef8d-4569-9295-b410b3057cc4 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:09.170341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fd2f41-ef8d-4569-9295-b410b3057cc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:09.170822 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fd2f41-ef8d-4569-9295-b410b3057cc4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:09.172204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fd2f41-ef8d-4569-9295-b410b3057cc4 None admin] RBAC: Adding query filter params (role.id=f425b98e3acf4e13b0da24b21e59cfce, scope.project.id=957c97404bb9448e9d03eee0d3225b85, user.id=229bce563b8c45fb8a24a170a209bf2f) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:09.172781 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fd2f41-ef8d-4569-9295-b410b3057cc4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:09.175649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fd2f41-ef8d-4569-9295-b410b3057cc4 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:09.193724 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 98/208] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:14:08 2022] GET /identity/v3/role_assignments?user.id=229bce563b8c45fb8a24a170a209bf2f&scope.project.id=957c97404bb9448e9d03eee0d3225b85&role.id=f425b98e3acf4e13b0da24b21e59cfce => generated 608 bytes in 352 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:14:10.755351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f552319d-67e1-48f0-a8c4-495fd73e469a None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:10.755351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f552319d-67e1-48f0-a8c4-495fd73e469a None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:10.755351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f552319d-67e1-48f0-a8c4-495fd73e469a None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:10.755351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 111/209] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:10.762505 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5da72d-66d6-469f-b2df-0208fc675bb8 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:10.763500 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5da72d-66d6-469f-b2df-0208fc675bb8 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:10.764000 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5da72d-66d6-469f-b2df-0208fc675bb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:10.785509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-3f5da72d-66d6-469f-b2df-0208fc675bb8 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:10.787332 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-3f5da72d-66d6-469f-b2df-0208fc675bb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:10.876923 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 99/210] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:10.882409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37afd976-91df-4947-86c4-333199de47c3 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:10.882689 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37afd976-91df-4947-86c4-333199de47c3 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:10.883169 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37afd976-91df-4947-86c4-333199de47c3 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:10.886608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 112/211] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:10.891897 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73574e6-a911-451f-94db-e90461608e9b None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:10.892507 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73574e6-a911-451f-94db-e90461608e9b None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:10.892934 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73574e6-a911-451f-94db-e90461608e9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:10.916694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-e73574e6-a911-451f-94db-e90461608e9b None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:10.918208 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-e73574e6-a911-451f-94db-e90461608e9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:10.970367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:11.018369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 100/212] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:11.029730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24633f17-f951-4a0d-9f82-f17b2df17672 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:11.030281 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24633f17-f951-4a0d-9f82-f17b2df17672 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:11.030703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24633f17-f951-4a0d-9f82-f17b2df17672 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:11.032560 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 113/213] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:11.039426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e017b1-65e0-4d3f-a83d-c2056b5d7bdb None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:11.039944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e017b1-65e0-4d3f-a83d-c2056b5d7bdb None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:11.040434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e017b1-65e0-4d3f-a83d-c2056b5d7bdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:11.061174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-01e017b1-65e0-4d3f-a83d-c2056b5d7bdb None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:11.062649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-01e017b1-65e0-4d3f-a83d-c2056b5d7bdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:11.076613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a51d86-a45b-46de-a4a1-3608c7645b68 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:11.077516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a51d86-a45b-46de-a4a1-3608c7645b68 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:11.078324 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a51d86-a45b-46de-a4a1-3608c7645b68 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:11.081169 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 114/214] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:11 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:14:11.091140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bd1cb8-b670-4cdd-85f3-63b08121cfa8 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:11.092027 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bd1cb8-b670-4cdd-85f3-63b08121cfa8 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:11.092817 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bd1cb8-b670-4cdd-85f3-63b08121cfa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:11.125673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-66bd1cb8-b670-4cdd-85f3-63b08121cfa8 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:11.128228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-66bd1cb8-b670-4cdd-85f3-63b08121cfa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:11.158057 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 101/215] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:11.184147 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e017b1-65e0-4d3f-a83d-c2056b5d7bdb None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:11.219260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 115/216] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:14:11.313206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e017b1-65e0-4d3f-a83d-c2056b5d7bdb None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:11.315473 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:11.356264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e2e128-3b1a-4b23-bcb2-dcdaf5f419db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:11.357337 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e2e128-3b1a-4b23-bcb2-dcdaf5f419db None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:11.357795 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e2e128-3b1a-4b23-bcb2-dcdaf5f419db None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:11.358076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e2e128-3b1a-4b23-bcb2-dcdaf5f419db None admin] PATH_INFO: `/v3/domains/default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:11.359353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e2e128-3b1a-4b23-bcb2-dcdaf5f419db None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:11.361163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e2e128-3b1a-4b23-bcb2-dcdaf5f419db None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:11.363746 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 102/217] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:14:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:11.372972 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bd1cb8-b670-4cdd-85f3-63b08121cfa8 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:11.417968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e2e128-3b1a-4b23-bcb2-dcdaf5f419db None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:11.457687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:11.538907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bd1cb8-b670-4cdd-85f3-63b08121cfa8 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:11.599797 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e2e128-3b1a-4b23-bcb2-dcdaf5f419db None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:11.606734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d7512e-6e9f-474b-bdaf-2163561cf158 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:11.610804 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d7512e-6e9f-474b-bdaf-2163561cf158 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:11.611222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d7512e-6e9f-474b-bdaf-2163561cf158 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:11.611613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d7512e-6e9f-474b-bdaf-2163561cf158 None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:11.612653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d7512e-6e9f-474b-bdaf-2163561cf158 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:11.615934 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d7512e-6e9f-474b-bdaf-2163561cf158 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:11.660351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 116/218] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:14:11 2022] POST /identity/v3/users => generated 297 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:14:11.660351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d7512e-6e9f-474b-bdaf-2163561cf158 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:11.685715 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d4eddd-272d-4c5c-9396-49fc36dd5125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:11.686372 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d4eddd-272d-4c5c-9396-49fc36dd5125 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:11.686650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d4eddd-272d-4c5c-9396-49fc36dd5125 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:11.686925 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d4eddd-272d-4c5c-9396-49fc36dd5125 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:11.700585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d4eddd-272d-4c5c-9396-49fc36dd5125 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:11.705615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-52d4eddd-272d-4c5c-9396-49fc36dd5125 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:11.705615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:11.705615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:11.705615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:11.705615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:11.705615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:11.705615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:11.705615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:11.705615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:11.705615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:11.705615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:11.705615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:11.705615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:11.705615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:11.705615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:11.705615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:11.707453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:11.707453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:11.707453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:11.707453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:11.707453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:11.707453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:11.707453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:11.707453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:11.707453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:11.707453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:11.707453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:11.707453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:11.707453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:11.707453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:11.707453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:11.707453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:11.709615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:11.709615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:11.709615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:11.709615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:11.709615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:11.709615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:11.709615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:11.709615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:11.709615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:11.709615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:11.709615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:11.709615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:11.709615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 103/219] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:11 2022] GET /identity/v3/roles/member => generated 84 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:14:11.709615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d4eddd-272d-4c5c-9396-49fc36dd5125 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:11.752009 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:11.845101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d4eddd-272d-4c5c-9396-49fc36dd5125 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:11.865971 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:11.873915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d7512e-6e9f-474b-bdaf-2163561cf158 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:11.896164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7717388c-7788-46bf-82f4-c328816899da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:11.896974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7717388c-7788-46bf-82f4-c328816899da None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:11.897249 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7717388c-7788-46bf-82f4-c328816899da None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:11.897530 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7717388c-7788-46bf-82f4-c328816899da None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:11.910459 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7717388c-7788-46bf-82f4-c328816899da None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:11.916082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-7717388c-7788-46bf-82f4-c328816899da None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:14:11.916082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:11.916082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:11.916082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:11.916082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:11.916082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:11.916082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:11.916082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:11.916082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:11.916082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:11.916082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:11.916082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:11.916082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:11.916082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:11.916082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:11.916082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:11.917445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:11.917445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:11.917445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:11.917445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:11.917445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:11.917445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:11.917445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:11.917445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:11.917445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:11.917445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:11.917445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:11.917445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:11.917445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:11.917445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:11.917445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:11.917445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:11.918875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:11.918875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:11.918875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:11.918875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:11.918875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:11.918875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:11.918875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:11.918875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:11.918875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:11.918875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:11.918875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:14:11.918875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:11.918875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 104/220] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:14:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:11.918875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7717388c-7788-46bf-82f4-c328816899da None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:11.966343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887cf806-839d-47be-8829-3167474e13df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:11.966343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887cf806-839d-47be-8829-3167474e13df None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:11.966343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887cf806-839d-47be-8829-3167474e13df None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:11.966343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887cf806-839d-47be-8829-3167474e13df None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:11.985437 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887cf806-839d-47be-8829-3167474e13df None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:11.996348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-887cf806-839d-47be-8829-3167474e13df None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:11.996348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:11.996348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:11.996348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:11.996348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:11.996348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:11.996348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:11.996348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:11.996348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:11.996348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:11.996348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:11.996348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:11.996348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:11.996348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:11.996348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:11.996348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:11.998050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:11.998050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:11.998050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:11.998050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:11.998050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:11.998050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:11.998050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:11.998050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:11.998050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:11.998050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:11.998050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:11.998050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:11.998050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:11.998050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:11.998050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:11.998050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:11.999445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:11.999445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:11.999445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:11.999445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:11.999445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:11.999445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:11.999445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:11.999445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:11.999445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:11.999445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:11.999445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:11.999445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:11.999445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 117/221] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:14:11 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 346 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:12.000792 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887cf806-839d-47be-8829-3167474e13df None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:12.070701 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:12.088422 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7717388c-7788-46bf-82f4-c328816899da None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:12.141432 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb0c2de-e52d-4c9b-82a0-6b093be393eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:12.142598 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb0c2de-e52d-4c9b-82a0-6b093be393eb None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:12.142598 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb0c2de-e52d-4c9b-82a0-6b093be393eb None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:12.142598 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb0c2de-e52d-4c9b-82a0-6b093be393eb None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:12.143196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb0c2de-e52d-4c9b-82a0-6b093be393eb None admin] RBAC: Adding query filter params (name=member) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:12.143498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb0c2de-e52d-4c9b-82a0-6b093be393eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:12.145070 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb0c2de-e52d-4c9b-82a0-6b093be393eb None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:12.158014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 105/222] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:11 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:12.159960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb0c2de-e52d-4c9b-82a0-6b093be393eb None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:12.201656 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887cf806-839d-47be-8829-3167474e13df None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:12.268562 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d13180f-963c-40be-b252-ce79fb1df2e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:12.269645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d13180f-963c-40be-b252-ce79fb1df2e7 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:12.270196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d13180f-963c-40be-b252-ce79fb1df2e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:12.270676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d13180f-963c-40be-b252-ce79fb1df2e7 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:12.272053 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d13180f-963c-40be-b252-ce79fb1df2e7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:12.272644 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d13180f-963c-40be-b252-ce79fb1df2e7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:12.275207 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d13180f-963c-40be-b252-ce79fb1df2e7 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:12.292370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 118/223] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:12.293883 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d13180f-963c-40be-b252-ce79fb1df2e7 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:12.328014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:12.364707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb0c2de-e52d-4c9b-82a0-6b093be393eb None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:12.440002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589ca160-cf28-479d-aa07-3a2b522483f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:12.441014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589ca160-cf28-479d-aa07-3a2b522483f9 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:12.441497 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589ca160-cf28-479d-aa07-3a2b522483f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:12.441987 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589ca160-cf28-479d-aa07-3a2b522483f9 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:12.442496 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d13180f-963c-40be-b252-ce79fb1df2e7 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:12.443266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589ca160-cf28-479d-aa07-3a2b522483f9 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:12.443805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589ca160-cf28-479d-aa07-3a2b522483f9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:12.446750 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589ca160-cf28-479d-aa07-3a2b522483f9 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:12.476378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 106/224] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:14:12 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:12.476378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589ca160-cf28-479d-aa07-3a2b522483f9 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:12.496873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f5dfdf-5606-4653-9da7-b0af9210aed3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:12.498469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f5dfdf-5606-4653-9da7-b0af9210aed3 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:12.498962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f5dfdf-5606-4653-9da7-b0af9210aed3 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:12.499451 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f5dfdf-5606-4653-9da7-b0af9210aed3 None admin] PATH_INFO: `/v3/users/75cd754f0a2b4f4b83d337bdeb2f2c36` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:12.507920 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:12.524953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f5dfdf-5606-4653-9da7-b0af9210aed3 None admin] RBAC: Authorizing `identity:get_user(user_id=75cd754f0a2b4f4b83d337bdeb2f2c36)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:12.527453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f5dfdf-5606-4653-9da7-b0af9210aed3 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:12.534969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 119/225] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:12 2022] GET /identity/v3/users/75cd754f0a2b4f4b83d337bdeb2f2c36 => generated 297 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:12.534969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f5dfdf-5606-4653-9da7-b0af9210aed3 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:12.653478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:12.686018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f5dfdf-5606-4653-9da7-b0af9210aed3 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:12.695001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589ca160-cf28-479d-aa07-3a2b522483f9 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:12.741120 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc3527d-225c-4683-88e1-816d2b29a171 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:12.742067 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc3527d-225c-4683-88e1-816d2b29a171 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:12.743093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc3527d-225c-4683-88e1-816d2b29a171 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:12.743593 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc3527d-225c-4683-88e1-816d2b29a171 None admin] PATH_INFO: `/v3/users/21ad892cf1a54f64933e32be81fa2dc1` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:12.745613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc3527d-225c-4683-88e1-816d2b29a171 None admin] RBAC: Authorizing `identity:get_user(user_id=21ad892cf1a54f64933e32be81fa2dc1)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:12.747665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc3527d-225c-4683-88e1-816d2b29a171 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:12.750348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 120/226] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:12 2022] GET /identity/v3/users/21ad892cf1a54f64933e32be81fa2dc1 => generated 283 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:12.751695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc3527d-225c-4683-88e1-816d2b29a171 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:12.764661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8386d321-acbe-42c9-a208-b2dcf217583f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:12.765646 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8386d321-acbe-42c9-a208-b2dcf217583f None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:12.766317 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8386d321-acbe-42c9-a208-b2dcf217583f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:12.766898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8386d321-acbe-42c9-a208-b2dcf217583f None admin] PATH_INFO: `/v3/users/21ad892cf1a54f64933e32be81fa2dc1` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:12.769173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8386d321-acbe-42c9-a208-b2dcf217583f None admin] RBAC: Authorizing `identity:get_user(user_id=21ad892cf1a54f64933e32be81fa2dc1)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:12.772911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8386d321-acbe-42c9-a208-b2dcf217583f None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:12.776555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 107/227] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:12 2022] GET /identity/v3/users/21ad892cf1a54f64933e32be81fa2dc1 => generated 283 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:12.778334 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8386d321-acbe-42c9-a208-b2dcf217583f None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:12.917595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:12.935080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc3527d-225c-4683-88e1-816d2b29a171 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:12.943819 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8386d321-acbe-42c9-a208-b2dcf217583f None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:12.984758 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b074d2ed-6cfe-47a5-b7e2-1e1343543d17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:12.985735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b074d2ed-6cfe-47a5-b7e2-1e1343543d17 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:12.986200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b074d2ed-6cfe-47a5-b7e2-1e1343543d17 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:12.986669 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b074d2ed-6cfe-47a5-b7e2-1e1343543d17 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:12.988077 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b074d2ed-6cfe-47a5-b7e2-1e1343543d17 None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:12.990660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b074d2ed-6cfe-47a5-b7e2-1e1343543d17 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:12.993349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 108/228] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:12 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:12.994402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b074d2ed-6cfe-47a5-b7e2-1e1343543d17 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:13.006329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e481273c-c90a-4ad0-bfad-3f8b52c14e39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:13.007472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e481273c-c90a-4ad0-bfad-3f8b52c14e39 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:13.008032 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e481273c-c90a-4ad0-bfad-3f8b52c14e39 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:13.008924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e481273c-c90a-4ad0-bfad-3f8b52c14e39 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:13.032060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:13.100995 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e481273c-c90a-4ad0-bfad-3f8b52c14e39 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:13.102789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e481273c-c90a-4ad0-bfad-3f8b52c14e39 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:13.124477 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b074d2ed-6cfe-47a5-b7e2-1e1343543d17 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:13.167123 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da3f0e6-98a6-4df6-a7a1-81964894378c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:13.168151 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da3f0e6-98a6-4df6-a7a1-81964894378c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:13.168648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da3f0e6-98a6-4df6-a7a1-81964894378c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:13.169171 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da3f0e6-98a6-4df6-a7a1-81964894378c None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:13.170780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da3f0e6-98a6-4df6-a7a1-81964894378c None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:13.172690 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da3f0e6-98a6-4df6-a7a1-81964894378c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:13.174846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 109/229] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:12 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:13.175950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da3f0e6-98a6-4df6-a7a1-81964894378c None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:13.203639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:13.230496 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 121/230] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 479 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:14:13.235513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e481273c-c90a-4ad0-bfad-3f8b52c14e39 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:13.292248 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:13.337476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da3f0e6-98a6-4df6-a7a1-81964894378c None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:13.366042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e481273c-c90a-4ad0-bfad-3f8b52c14e39 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:13.406468 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd58a9fc-a2d1-4a6d-81f8-e41ea568a5f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:13.407426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd58a9fc-a2d1-4a6d-81f8-e41ea568a5f1 None admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:13.408223 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd58a9fc-a2d1-4a6d-81f8-e41ea568a5f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:13.408564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd58a9fc-a2d1-4a6d-81f8-e41ea568a5f1 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535/users/21ad892cf1a54f64933e32be81fa2dc1/roles/e114d28e2dd147fcb39452dd461791c0` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:13.411607 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd58a9fc-a2d1-4a6d-81f8-e41ea568a5f1 None admin] RBAC: Authorizing `identity:create_grant(project_id=2728b0ff765c4e9fa2c0cb33c62b0535, user_id=21ad892cf1a54f64933e32be81fa2dc1, role_id=e114d28e2dd147fcb39452dd461791c0)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:13.414039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd58a9fc-a2d1-4a6d-81f8-e41ea568a5f1 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:13.418564 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b69e35f-63dd-4a28-a648-0233620b8057 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:13.419271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b69e35f-63dd-4a28-a648-0233620b8057 None admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:13.419582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b69e35f-63dd-4a28-a648-0233620b8057 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:13.420117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b69e35f-63dd-4a28-a648-0233620b8057 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535/users/21ad892cf1a54f64933e32be81fa2dc1/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:13.422848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b69e35f-63dd-4a28-a648-0233620b8057 None admin] RBAC: Authorizing `identity:create_grant(project_id=2728b0ff765c4e9fa2c0cb33c62b0535, user_id=21ad892cf1a54f64933e32be81fa2dc1, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:13.424395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b69e35f-63dd-4a28-a648-0233620b8057 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:13.433386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 110/231] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:14:13 2022] PUT /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535/users/21ad892cf1a54f64933e32be81fa2dc1/roles/e114d28e2dd147fcb39452dd461791c0 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:14:13.440481 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd58a9fc-a2d1-4a6d-81f8-e41ea568a5f1 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:13.452348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 122/232] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:14:13 2022] PUT /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535/users/21ad892cf1a54f64933e32be81fa2dc1/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:14:13.607416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:13.649752 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd58a9fc-a2d1-4a6d-81f8-e41ea568a5f1 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:13.711826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cf1fc2-4d44-4aac-adfe-b4794edbd6f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:13.712575 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cf1fc2-4d44-4aac-adfe-b4794edbd6f6 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:13.712866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cf1fc2-4d44-4aac-adfe-b4794edbd6f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:13.713143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cf1fc2-4d44-4aac-adfe-b4794edbd6f6 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:13.714546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cf1fc2-4d44-4aac-adfe-b4794edbd6f6 None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:13.716140 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cf1fc2-4d44-4aac-adfe-b4794edbd6f6 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:13.718046 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 111/233] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:13 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:13.724563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b69e35f-63dd-4a28-a648-0233620b8057 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:13.900744 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b69e35f-63dd-4a28-a648-0233620b8057 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:13.916909 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:13.968149 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c373c43f-30f6-4f76-ac17-55adaf16cc72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:13.969847 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c373c43f-30f6-4f76-ac17-55adaf16cc72 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:13.970374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c373c43f-30f6-4f76-ac17-55adaf16cc72 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:13.970839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c373c43f-30f6-4f76-ac17-55adaf16cc72 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:13.971868 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c373c43f-30f6-4f76-ac17-55adaf16cc72 None admin] RBAC: Adding query filter params (role.id=bc6aeaa495ca4f3dbe72905c744c2233, scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535, user.id=75cd754f0a2b4f4b83d337bdeb2f2c36) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:13.972401 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c373c43f-30f6-4f76-ac17-55adaf16cc72 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:13.976682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c373c43f-30f6-4f76-ac17-55adaf16cc72 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:13.990182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 123/234] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:14:13 2022] GET /identity/v3/role_assignments?user.id=75cd754f0a2b4f4b83d337bdeb2f2c36&scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535&role.id=bc6aeaa495ca4f3dbe72905c744c2233 => generated 259 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:14:14.823682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0557b6a2-d00b-434f-aaf4-e32fd11a1380 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:14.824462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0557b6a2-d00b-434f-aaf4-e32fd11a1380 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:14.824595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0557b6a2-d00b-434f-aaf4-e32fd11a1380 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:14.827034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 112/235] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:14 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:14.837522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fc8b94-77b2-4b7b-8020-39a4191900f6 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:14.838463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fc8b94-77b2-4b7b-8020-39a4191900f6 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:14.839309 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fc8b94-77b2-4b7b-8020-39a4191900f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:14.885785 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-00fc8b94-77b2-4b7b-8020-39a4191900f6 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:14.888526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-00fc8b94-77b2-4b7b-8020-39a4191900f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:14.993895 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1494ddb7-f9f0-43c8-a4cf-289e3931ee52 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:14.994192 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1494ddb7-f9f0-43c8-a4cf-289e3931ee52 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:14.994517 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1494ddb7-f9f0-43c8-a4cf-289e3931ee52 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:14.996122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 113/236] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:15.006246 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d62eeb-aae5-4fcf-b656-a50d29d029dc None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:15.006650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d62eeb-aae5-4fcf-b656-a50d29d029dc None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:15.006826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d62eeb-aae5-4fcf-b656-a50d29d029dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:15.034134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-33d62eeb-aae5-4fcf-b656-a50d29d029dc None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:15.036843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-33d62eeb-aae5-4fcf-b656-a50d29d029dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:15.118607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 124/237] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:15.221547 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 114/238] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:15.620185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fc8b94-77b2-4b7b-8020-39a4191900f6 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:15.885522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:15.891937 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fc8b94-77b2-4b7b-8020-39a4191900f6 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:15.955455 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d62eeb-aae5-4fcf-b656-a50d29d029dc None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:15.980020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf049de-203e-44b4-9d60-0aac7d53552b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:15.981646 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf049de-203e-44b4-9d60-0aac7d53552b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:15.982492 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf049de-203e-44b4-9d60-0aac7d53552b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:15.983256 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf049de-203e-44b4-9d60-0aac7d53552b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:16.021780 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf049de-203e-44b4-9d60-0aac7d53552b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:16.033231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-8cf049de-203e-44b4-9d60-0aac7d53552b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:14:16.033231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:16.033231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:16.033231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:16.033231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:16.033231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:16.033231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:16.033231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:16.033231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:16.033231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:16.033231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:16.033231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:16.033231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:16.033231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:16.033231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:16.033231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:16.035043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:16.035043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:16.035043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:16.035043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:16.035043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:16.035043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:16.035043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:16.035043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:16.035043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:16.035043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:16.035043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:16.035043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:16.035043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:16.035043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:16.035043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:16.035043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:16.037845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:16.037845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:16.037845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:16.037845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:16.037845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:16.037845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:16.037845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:16.037845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:16.037845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:16.037845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:16.037845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:14:16.037845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:16.042042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 125/239] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:15 2022] GET /identity/v3/roles/reader => generated 84 bytes in 423 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:16.051106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf049de-203e-44b4-9d60-0aac7d53552b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:16.208032 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:16.251334 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d62eeb-aae5-4fcf-b656-a50d29d029dc None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:16.325389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf049de-203e-44b4-9d60-0aac7d53552b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:16.375676 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f313c578-13cc-4599-87aa-0f2e1bd5cd13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:16.376921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f313c578-13cc-4599-87aa-0f2e1bd5cd13 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:16.377455 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f313c578-13cc-4599-87aa-0f2e1bd5cd13 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:16.377892 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f313c578-13cc-4599-87aa-0f2e1bd5cd13 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:16.390278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f313c578-13cc-4599-87aa-0f2e1bd5cd13 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:16.397189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b1565b-87e5-4378-a0f5-94e035e19d9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:16.398237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b1565b-87e5-4378-a0f5-94e035e19d9d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:16.398700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b1565b-87e5-4378-a0f5-94e035e19d9d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:16.399164 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b1565b-87e5-4378-a0f5-94e035e19d9d None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:16.400471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b1565b-87e5-4378-a0f5-94e035e19d9d None admin] RBAC: Adding query filter params (name=reader) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:16.401010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b1565b-87e5-4378-a0f5-94e035e19d9d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:16.401173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f313c578-13cc-4599-87aa-0f2e1bd5cd13 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:16.403612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b1565b-87e5-4378-a0f5-94e035e19d9d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:16.404344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 115/240] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:14:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 450 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:16.410938 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f313c578-13cc-4599-87aa-0f2e1bd5cd13 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:16.423021 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 126/241] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:16 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 372 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:14:16.432424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b1565b-87e5-4378-a0f5-94e035e19d9d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:16.533025 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:16.731793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f313c578-13cc-4599-87aa-0f2e1bd5cd13 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:16.738851 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b1565b-87e5-4378-a0f5-94e035e19d9d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:16.864728 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7441037-8bd8-4f73-9641-3ad40d430ff5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:16.865845 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7441037-8bd8-4f73-9641-3ad40d430ff5 None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:16.866353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7441037-8bd8-4f73-9641-3ad40d430ff5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:16.866844 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7441037-8bd8-4f73-9641-3ad40d430ff5 None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:16.868087 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7441037-8bd8-4f73-9641-3ad40d430ff5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:16.870871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7441037-8bd8-4f73-9641-3ad40d430ff5 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:16.878824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f505d0-c6a3-43f1-86bb-56b93fe419ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:16.879943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f505d0-c6a3-43f1-86bb-56b93fe419ba None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:16.880384 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f505d0-c6a3-43f1-86bb-56b93fe419ba None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:16.880885 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f505d0-c6a3-43f1-86bb-56b93fe419ba None admin] PATH_INFO: `/v3/users/167ee884b89641bfa8a5cc73e6d7267d` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:16.898824 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:16.912421 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 116/242] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:14:16 2022] POST /identity/v3/users => generated 293 bytes in 496 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:14:16.913378 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f505d0-c6a3-43f1-86bb-56b93fe419ba None admin] RBAC: Authorizing `identity:get_user(user_id=167ee884b89641bfa8a5cc73e6d7267d)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:16.913940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea78121-a741-4985-b85f-92dc3b85e668 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:16.914233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea78121-a741-4985-b85f-92dc3b85e668 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:16.914512 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea78121-a741-4985-b85f-92dc3b85e668 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:16.916245 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 117/243] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:16.916461 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f505d0-c6a3-43f1-86bb-56b93fe419ba None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:16.919445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 127/244] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:16 2022] GET /identity/v3/users/167ee884b89641bfa8a5cc73e6d7267d => generated 297 bytes in 488 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:14:16.929609 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f28eb12-a557-4a51-b0ba-c37f4cb6f016 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:16.929909 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f505d0-c6a3-43f1-86bb-56b93fe419ba None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:16.930045 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f28eb12-a557-4a51-b0ba-c37f4cb6f016 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:16.930181 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f28eb12-a557-4a51-b0ba-c37f4cb6f016 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:16.968790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-0f28eb12-a557-4a51-b0ba-c37f4cb6f016 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:16.970053 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0f28eb12-a557-4a51-b0ba-c37f4cb6f016 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:17.065004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 118/245] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:17.142725 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:17.176991 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f505d0-c6a3-43f1-86bb-56b93fe419ba None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:17.202087 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00005af0-ebb2-4210-abf7-b06184e29837 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:17.202562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00005af0-ebb2-4210-abf7-b06184e29837 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:17.203068 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00005af0-ebb2-4210-abf7-b06184e29837 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:17.206533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 119/246] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:17 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:14:17.220998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd11457-95fa-4a37-a5cb-ff778a3bbad5 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:17.221470 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd11457-95fa-4a37-a5cb-ff778a3bbad5 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:17.221930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd11457-95fa-4a37-a5cb-ff778a3bbad5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:17.263908 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80dc829e-d93f-4b72-8880-f9958c6814b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:17.265111 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dc829e-d93f-4b72-8880-f9958c6814b1 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:17.265704 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dc829e-d93f-4b72-8880-f9958c6814b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:17.266206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dc829e-d93f-4b72-8880-f9958c6814b1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:17.268699 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-ebd11457-95fa-4a37-a5cb-ff778a3bbad5 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:17.275490 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ebd11457-95fa-4a37-a5cb-ff778a3bbad5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:17.404147 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 120/247] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:14:17.507731 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f21176-e1b1-49ce-8791-e7bd30a58710 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:17.512342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f21176-e1b1-49ce-8791-e7bd30a58710 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:17.512342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f21176-e1b1-49ce-8791-e7bd30a58710 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:17.516337 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 121/248] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:17 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:17.530570 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffb3f7b-3e04-4a74-b70f-c8e553c329ab None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:17.536353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffb3f7b-3e04-4a74-b70f-c8e553c329ab None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:17.536353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffb3f7b-3e04-4a74-b70f-c8e553c329ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:17.586272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-1ffb3f7b-3e04-4a74-b70f-c8e553c329ab None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:17.592345 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-1ffb3f7b-3e04-4a74-b70f-c8e553c329ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:17.592542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80dc829e-d93f-4b72-8880-f9958c6814b1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:17.592542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80dc829e-d93f-4b72-8880-f9958c6814b1 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:17.730451 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:17.753254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 122/249] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:17.754577 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffb3f7b-3e04-4a74-b70f-c8e553c329ab None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:17.794912 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:17.835599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 128/250] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 907 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:14:17.838899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80dc829e-d93f-4b72-8880-f9958c6814b1 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:17.954038 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffb3f7b-3e04-4a74-b70f-c8e553c329ab None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:17.999735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e71a9c6-8a23-4f17-ae05-4c14220c63fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:18.001409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e71a9c6-8a23-4f17-ae05-4c14220c63fb None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:18.001907 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e71a9c6-8a23-4f17-ae05-4c14220c63fb None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:18.002391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e71a9c6-8a23-4f17-ae05-4c14220c63fb None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:18.017215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e71a9c6-8a23-4f17-ae05-4c14220c63fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:18.023972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-8e71a9c6-8a23-4f17-ae05-4c14220c63fb None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:18.023972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:18.023972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:18.023972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:18.023972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:18.023972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:18.023972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:18.023972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:18.023972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:18.023972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:18.023972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:18.023972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:18.023972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:18.023972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:18.023972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:18.023972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:18.025164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:18.025164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:18.025164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:18.025164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:18.025164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:18.025164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:18.025164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:18.025164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:18.025164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:18.025164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:18.025164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:18.025164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:18.025164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:18.025164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:18.025164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:18.025164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:18.025974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:18.025974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:18.025974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:18.025974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:18.025974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:18.025974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:18.025974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:18.025974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:18.025974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:18.025974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:18.025974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:18.025974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:18.028592 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 123/251] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:14:17 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:18.030784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e71a9c6-8a23-4f17-ae05-4c14220c63fb None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:18.050483 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80dc829e-d93f-4b72-8880-f9958c6814b1 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:18.091884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb740f78-5819-4165-9cea-de178813c98a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:18.092561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb740f78-5819-4165-9cea-de178813c98a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:18.092840 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb740f78-5819-4165-9cea-de178813c98a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:18.093115 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb740f78-5819-4165-9cea-de178813c98a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:18.106231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb740f78-5819-4165-9cea-de178813c98a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:18.111291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-cb740f78-5819-4165-9cea-de178813c98a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:18.111291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:18.111291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:18.111291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:18.111291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:18.111291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:18.111291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:18.111291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:18.111291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:18.111291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:18.111291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:18.111291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:18.111291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:18.111291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:18.111291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:18.111291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:18.112269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:18.112269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:18.112269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:18.112269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:18.112269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:18.112269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:18.112269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:18.112269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:18.112269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:18.112269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:18.112269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:18.112269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:18.112269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:18.112269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:18.112269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:18.112269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:18.113117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:18.113117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:18.113117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:18.113117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:18.113117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:18.113117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:18.113117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:18.113117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:18.113117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:18.113117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:18.113117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:18.113117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:18.113117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 129/252] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:18.114753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb740f78-5819-4165-9cea-de178813c98a None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:18.129912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:18.167417 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:18.233591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e71a9c6-8a23-4f17-ae05-4c14220c63fb None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:18.238022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb740f78-5819-4165-9cea-de178813c98a None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:18.280241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a09dcac-303a-46ca-9882-1c07724a20d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:18.281043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a09dcac-303a-46ca-9882-1c07724a20d4 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:18.281520 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a09dcac-303a-46ca-9882-1c07724a20d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:18.281871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a09dcac-303a-46ca-9882-1c07724a20d4 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:18.283139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a09dcac-303a-46ca-9882-1c07724a20d4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:18.283617 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a09dcac-303a-46ca-9882-1c07724a20d4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:18.284917 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63280660-b5ce-40c1-8171-b7db55335493 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:18.286264 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a09dcac-303a-46ca-9882-1c07724a20d4 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:18.286673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63280660-b5ce-40c1-8171-b7db55335493 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:18.287177 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63280660-b5ce-40c1-8171-b7db55335493 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:18.287608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63280660-b5ce-40c1-8171-b7db55335493 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:18.290054 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63280660-b5ce-40c1-8171-b7db55335493 None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:18.291767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63280660-b5ce-40c1-8171-b7db55335493 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:18.294840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 124/253] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:18 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:18.295947 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63280660-b5ce-40c1-8171-b7db55335493 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:18.297109 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 130/254] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:14:18 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:18.302743 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a09dcac-303a-46ca-9882-1c07724a20d4 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:18.453244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63280660-b5ce-40c1-8171-b7db55335493 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:18.454396 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a09dcac-303a-46ca-9882-1c07724a20d4 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:18.456763 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:18.487975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:18.500045 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc480c1-a1b3-4cd2-8e67-bdef143f5841 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:18.500730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc480c1-a1b3-4cd2-8e67-bdef143f5841 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:18.501011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc480c1-a1b3-4cd2-8e67-bdef143f5841 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:18.501300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc480c1-a1b3-4cd2-8e67-bdef143f5841 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:18.515041 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc480c1-a1b3-4cd2-8e67-bdef143f5841 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:18.515447 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def263d1-9710-4e83-88bc-240f22c515c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:18.516030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def263d1-9710-4e83-88bc-240f22c515c5 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:18.516357 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def263d1-9710-4e83-88bc-240f22c515c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:18.516616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def263d1-9710-4e83-88bc-240f22c515c5 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:18.517315 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def263d1-9710-4e83-88bc-240f22c515c5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:18.517623 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def263d1-9710-4e83-88bc-240f22c515c5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:18.519061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def263d1-9710-4e83-88bc-240f22c515c5 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:18.520893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-1bc480c1-a1b3-4cd2-8e67-bdef143f5841 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:14:18.520893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:18.520893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:18.520893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:18.520893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:18.520893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:18.520893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:18.520893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:18.520893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:18.520893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:18.520893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:18.520893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:18.520893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:18.520893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:18.520893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:18.520893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:18.523177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:18.523177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:18.523177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:18.523177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:18.523177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:18.523177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:18.523177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:18.523177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:18.523177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:18.523177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:18.523177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:18.523177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:18.523177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:18.523177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:18.523177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:18.523177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:18.524889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:18.524889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:18.524889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:18.524889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:18.524889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:18.524889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:18.524889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:18.524889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:18.524889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:18.524889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:18.524889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:14:18.524889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:18.524889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 131/255] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:14:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:18.524889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc480c1-a1b3-4cd2-8e67-bdef143f5841 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:18.531263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 125/256] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:18.531870 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def263d1-9710-4e83-88bc-240f22c515c5 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:18.647268 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc480c1-a1b3-4cd2-8e67-bdef143f5841 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:18.691339 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af688464-21af-4805-92f6-ae2f6a4caca6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:18.692057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af688464-21af-4805-92f6-ae2f6a4caca6 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:18.692358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af688464-21af-4805-92f6-ae2f6a4caca6 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:18.692640 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af688464-21af-4805-92f6-ae2f6a4caca6 None admin] PATH_INFO: `/v3/users/21ad892cf1a54f64933e32be81fa2dc1` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:18.693270 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def263d1-9710-4e83-88bc-240f22c515c5 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:18.693909 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af688464-21af-4805-92f6-ae2f6a4caca6 None admin] RBAC: Authorizing `identity:get_user(user_id=21ad892cf1a54f64933e32be81fa2dc1)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:18.695792 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af688464-21af-4805-92f6-ae2f6a4caca6 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:18.704025 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 132/257] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:18 2022] GET /identity/v3/users/21ad892cf1a54f64933e32be81fa2dc1 => generated 283 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:18.705261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af688464-21af-4805-92f6-ae2f6a4caca6 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:18.738447 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244dfeb3-edcb-4597-8dc2-a0656ac3bdf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:18.739389 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244dfeb3-edcb-4597-8dc2-a0656ac3bdf5 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:18.739772 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244dfeb3-edcb-4597-8dc2-a0656ac3bdf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:18.740150 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244dfeb3-edcb-4597-8dc2-a0656ac3bdf5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:18.741341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244dfeb3-edcb-4597-8dc2-a0656ac3bdf5 None admin] RBAC: Adding query filter params (role.id=f425b98e3acf4e13b0da24b21e59cfce, scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535, user.id=167ee884b89641bfa8a5cc73e6d7267d) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:18.741850 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244dfeb3-edcb-4597-8dc2-a0656ac3bdf5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:18.743459 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244dfeb3-edcb-4597-8dc2-a0656ac3bdf5 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:18.751112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 126/258] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:14:18 2022] GET /identity/v3/role_assignments?user.id=167ee884b89641bfa8a5cc73e6d7267d&scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535&role.id=f425b98e3acf4e13b0da24b21e59cfce => generated 259 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:18.752132 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:18.753397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244dfeb3-edcb-4597-8dc2-a0656ac3bdf5 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:18.817889 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:18.872353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af688464-21af-4805-92f6-ae2f6a4caca6 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:18.908859 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244dfeb3-edcb-4597-8dc2-a0656ac3bdf5 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:18.928843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc39b2c9-316e-4d16-b9df-dbd8881ef56b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:18.931849 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc39b2c9-316e-4d16-b9df-dbd8881ef56b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:18.932176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc39b2c9-316e-4d16-b9df-dbd8881ef56b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:18.933730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc39b2c9-316e-4d16-b9df-dbd8881ef56b None admin] PATH_INFO: `/v3/users/75cd754f0a2b4f4b83d337bdeb2f2c36` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:18.935798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc39b2c9-316e-4d16-b9df-dbd8881ef56b None admin] RBAC: Authorizing `identity:get_user(user_id=75cd754f0a2b4f4b83d337bdeb2f2c36)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:18.939237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc39b2c9-316e-4d16-b9df-dbd8881ef56b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:18.943016 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 133/259] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:18 2022] GET /identity/v3/users/75cd754f0a2b4f4b83d337bdeb2f2c36 => generated 297 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:18.943692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc39b2c9-316e-4d16-b9df-dbd8881ef56b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:18.968803 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a66b64-2a29-469f-8730-df932126a598 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:18.969897 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a66b64-2a29-469f-8730-df932126a598 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:18.970397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a66b64-2a29-469f-8730-df932126a598 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:18.970880 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a66b64-2a29-469f-8730-df932126a598 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:18.973208 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a66b64-2a29-469f-8730-df932126a598 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:18.973823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a66b64-2a29-469f-8730-df932126a598 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:18.975499 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a66b64-2a29-469f-8730-df932126a598 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:18.987716 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 127/260] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:18.989076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a66b64-2a29-469f-8730-df932126a598 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:19.133331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a66b64-2a29-469f-8730-df932126a598 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:19.147036 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:19.213440 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:19.223546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c870d42f-5958-4ac5-9098-b1c4a6b58bd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:19.224429 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c870d42f-5958-4ac5-9098-b1c4a6b58bd7 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:19.224589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc39b2c9-316e-4d16-b9df-dbd8881ef56b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:19.224757 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c870d42f-5958-4ac5-9098-b1c4a6b58bd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:19.225043 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c870d42f-5958-4ac5-9098-b1c4a6b58bd7 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:19.233668 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c870d42f-5958-4ac5-9098-b1c4a6b58bd7 None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:19.235289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c870d42f-5958-4ac5-9098-b1c4a6b58bd7 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:19.237562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 128/261] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:18 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:19.238640 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c870d42f-5958-4ac5-9098-b1c4a6b58bd7 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:19.322889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43da5430-d545-4325-9717-5363c42c9577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:19.323962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43da5430-d545-4325-9717-5363c42c9577 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:19.324471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43da5430-d545-4325-9717-5363c42c9577 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:19.324965 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43da5430-d545-4325-9717-5363c42c9577 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:19.498711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43da5430-d545-4325-9717-5363c42c9577 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:19.500666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43da5430-d545-4325-9717-5363c42c9577 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:19.503396 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c870d42f-5958-4ac5-9098-b1c4a6b58bd7 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:19.614115 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb7f44e-a796-4c74-8d69-d0c4b1890138 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:19.614815 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb7f44e-a796-4c74-8d69-d0c4b1890138 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:19.615288 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb7f44e-a796-4c74-8d69-d0c4b1890138 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:19.615751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb7f44e-a796-4c74-8d69-d0c4b1890138 None admin] PATH_INFO: `/v3/users/21ad892cf1a54f64933e32be81fa2dc1` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:19.621634 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb7f44e-a796-4c74-8d69-d0c4b1890138 None admin] RBAC: Authorizing `identity:get_user(user_id=21ad892cf1a54f64933e32be81fa2dc1)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:19.624193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb7f44e-a796-4c74-8d69-d0c4b1890138 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:19.627141 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 129/262] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:19 2022] GET /identity/v3/users/21ad892cf1a54f64933e32be81fa2dc1 => generated 283 bytes in 389 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:19.628512 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb7f44e-a796-4c74-8d69-d0c4b1890138 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:19.654074 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:19.666085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 134/263] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 723 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:14:19.667948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43da5430-d545-4325-9717-5363c42c9577 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:19.824819 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:19.877346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43da5430-d545-4325-9717-5363c42c9577 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:19.952760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ffd82f-2f82-4a7e-b9c1-32c9dae65de4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:19.954202 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ffd82f-2f82-4a7e-b9c1-32c9dae65de4 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:19.954994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ffd82f-2f82-4a7e-b9c1-32c9dae65de4 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:19.955781 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ffd82f-2f82-4a7e-b9c1-32c9dae65de4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:20.023884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb7f44e-a796-4c74-8d69-d0c4b1890138 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:20.109398 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:20.120002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ffd82f-2f82-4a7e-b9c1-32c9dae65de4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:20.123312 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ffd82f-2f82-4a7e-b9c1-32c9dae65de4 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:20.169712 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5faf71a-6730-45bc-babf-957a22b99638 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:20.170550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5faf71a-6730-45bc-babf-957a22b99638 None admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:20.170691 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5faf71a-6730-45bc-babf-957a22b99638 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:20.171077 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5faf71a-6730-45bc-babf-957a22b99638 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535/users/75cd754f0a2b4f4b83d337bdeb2f2c36/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:20.173120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5faf71a-6730-45bc-babf-957a22b99638 None admin] RBAC: Authorizing `identity:create_grant(project_id=2728b0ff765c4e9fa2c0cb33c62b0535, user_id=75cd754f0a2b4f4b83d337bdeb2f2c36, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:20.174654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5faf71a-6730-45bc-babf-957a22b99638 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:20.214017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 130/264] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:14:19 2022] PUT /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535/users/75cd754f0a2b4f4b83d337bdeb2f2c36/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 586 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:14:20.215529 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5faf71a-6730-45bc-babf-957a22b99638 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:20.291721 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 135/265] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 625 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:14:20.298282 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ffd82f-2f82-4a7e-b9c1-32c9dae65de4 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:20.529299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5faf71a-6730-45bc-babf-957a22b99638 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:20.531542 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:20.553970 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:20.562158 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ffd82f-2f82-4a7e-b9c1-32c9dae65de4 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:20.620766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a72adb-3266-4fd4-a142-f5915fa4f170 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:20.621440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a72adb-3266-4fd4-a142-f5915fa4f170 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:20.621723 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a72adb-3266-4fd4-a142-f5915fa4f170 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:20.622005 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a72adb-3266-4fd4-a142-f5915fa4f170 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:20.624987 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a72adb-3266-4fd4-a142-f5915fa4f170 None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:20.626492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a72adb-3266-4fd4-a142-f5915fa4f170 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:20.628340 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 131/266] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:20 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 414 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:20.629668 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b87dc7-219b-4d37-8b58-a0795df79980 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:20.630140 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b87dc7-219b-4d37-8b58-a0795df79980 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:20.630423 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b87dc7-219b-4d37-8b58-a0795df79980 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:20.631825 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 132/267] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:20.640213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b87dc7-219b-4d37-8b58-a0795df79980 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:20.644297 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0585a20a-9ce8-473a-8876-4149d04b1f9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:20.645665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0585a20a-9ce8-473a-8876-4149d04b1f9e None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:20.645954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0585a20a-9ce8-473a-8876-4149d04b1f9e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:20.646223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0585a20a-9ce8-473a-8876-4149d04b1f9e None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:20.647498 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0585a20a-9ce8-473a-8876-4149d04b1f9e None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:20.649613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0585a20a-9ce8-473a-8876-4149d04b1f9e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:20.651740 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 136/268] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:20 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 354 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:14:20.654327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee89e882-e670-449b-a1eb-a82fe4704f04 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:20.654923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee89e882-e670-449b-a1eb-a82fe4704f04 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:20.655137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee89e882-e670-449b-a1eb-a82fe4704f04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:20.684713 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-ee89e882-e670-449b-a1eb-a82fe4704f04 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:20.686056 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ee89e882-e670-449b-a1eb-a82fe4704f04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:20.772506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 137/269] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:20.773323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee89e882-e670-449b-a1eb-a82fe4704f04 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:20.893496 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b87dc7-219b-4d37-8b58-a0795df79980 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:20.908556 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:20.916700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee89e882-e670-449b-a1eb-a82fe4704f04 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:20.958605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06fa4d5-b5d2-4d4b-9acd-db610a7585da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:20.959232 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06fa4d5-b5d2-4d4b-9acd-db610a7585da None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:20.959511 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06fa4d5-b5d2-4d4b-9acd-db610a7585da None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:20.959786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06fa4d5-b5d2-4d4b-9acd-db610a7585da None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:20.960565 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06fa4d5-b5d2-4d4b-9acd-db610a7585da None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535, user.id=21ad892cf1a54f64933e32be81fa2dc1) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:20.960876 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06fa4d5-b5d2-4d4b-9acd-db610a7585da None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:20.962560 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06fa4d5-b5d2-4d4b-9acd-db610a7585da None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:20.970010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 138/270] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:14:20 2022] GET /identity/v3/role_assignments?user.id=21ad892cf1a54f64933e32be81fa2dc1&scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 608 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:21.007361 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d743f1-7251-44e9-a3a8-0402d8cdbabf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:21.008477 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d743f1-7251-44e9-a3a8-0402d8cdbabf None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:21.008976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d743f1-7251-44e9-a3a8-0402d8cdbabf None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:21.009443 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d743f1-7251-44e9-a3a8-0402d8cdbabf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:21.010951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d743f1-7251-44e9-a3a8-0402d8cdbabf None admin] RBAC: Adding query filter params (role.id=e114d28e2dd147fcb39452dd461791c0, scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535, user.id=21ad892cf1a54f64933e32be81fa2dc1) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:21.011414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d743f1-7251-44e9-a3a8-0402d8cdbabf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:21.014259 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d743f1-7251-44e9-a3a8-0402d8cdbabf None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:21.033785 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 133/271] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:14:20 2022] GET /identity/v3/role_assignments?user.id=21ad892cf1a54f64933e32be81fa2dc1&scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535&role.id=e114d28e2dd147fcb39452dd461791c0 => generated 608 bytes in 394 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:21.306497 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06fa4d5-b5d2-4d4b-9acd-db610a7585da None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:21.393426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96700663-6d3f-499e-a397-b86f4c39be68 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:21.394531 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96700663-6d3f-499e-a397-b86f4c39be68 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:21.395216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96700663-6d3f-499e-a397-b86f4c39be68 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:21.397136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 134/272] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:21 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:14:21.404777 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafec6b4-132f-4252-b1ec-959f25343f1d None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:21.405408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafec6b4-132f-4252-b1ec-959f25343f1d None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:21.405920 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafec6b4-132f-4252-b1ec-959f25343f1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:21.451432 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-eafec6b4-132f-4252-b1ec-959f25343f1d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:21.452433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-eafec6b4-132f-4252-b1ec-959f25343f1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:21.526114 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:21.544134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06fa4d5-b5d2-4d4b-9acd-db610a7585da None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:21.590865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 135/273] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:14:21.623398 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04be6132-eb07-45cb-8aed-25d9bd595950 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:21.624521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04be6132-eb07-45cb-8aed-25d9bd595950 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:21.625009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04be6132-eb07-45cb-8aed-25d9bd595950 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:21.625500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04be6132-eb07-45cb-8aed-25d9bd595950 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:21.627698 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04be6132-eb07-45cb-8aed-25d9bd595950 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:21.630961 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04be6132-eb07-45cb-8aed-25d9bd595950 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:21.634531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 139/274] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:14:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:21.643653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafec6b4-132f-4252-b1ec-959f25343f1d None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:21.791238 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafec6b4-132f-4252-b1ec-959f25343f1d None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:21.847157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a9d835-3275-474d-9603-5e803b35e422 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:21.848593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a9d835-3275-474d-9603-5e803b35e422 None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:21.849153 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a9d835-3275-474d-9603-5e803b35e422 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:21.849643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a9d835-3275-474d-9603-5e803b35e422 None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:21.850512 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a9d835-3275-474d-9603-5e803b35e422 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:21.852244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a9d835-3275-474d-9603-5e803b35e422 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:21.885595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 136/275] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:14:21 2022] POST /identity/v3/users => generated 293 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:14:21.958687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04be6132-eb07-45cb-8aed-25d9bd595950 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:21.997264 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:22.116791 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04be6132-eb07-45cb-8aed-25d9bd595950 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:22.199874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61a3d97-6b86-49d9-a05c-95bd0b050ec2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:22.201089 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61a3d97-6b86-49d9-a05c-95bd0b050ec2 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:22.201517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61a3d97-6b86-49d9-a05c-95bd0b050ec2 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:22.201963 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61a3d97-6b86-49d9-a05c-95bd0b050ec2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:22.223377 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61a3d97-6b86-49d9-a05c-95bd0b050ec2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:22.232455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-e61a3d97-6b86-49d9-a05c-95bd0b050ec2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:22.232455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:22.232455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:22.232455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:22.232455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:22.232455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:22.232455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:22.232455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:22.232455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:22.232455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:22.232455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:22.232455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:22.232455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:22.232455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:22.232455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:22.232455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:22.234194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:22.234194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:22.234194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:22.234194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:22.234194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:22.234194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:22.234194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:22.234194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:22.234194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:22.234194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:22.234194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:22.234194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:22.234194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:22.234194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:22.234194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:22.234194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:22.235358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:22.235358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:22.235358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:22.235358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:22.235358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:22.235358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:22.235358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:22.235358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:22.235358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:22.235358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:22.235358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:22.235358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:22.235358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 140/276] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:22.242174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a9d835-3275-474d-9603-5e803b35e422 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:22.295868 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:22.375546 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff75760f-0ce7-45d7-bcf1-517cb412a94d None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:22.376581 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff75760f-0ce7-45d7-bcf1-517cb412a94d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:22.377305 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff75760f-0ce7-45d7-bcf1-517cb412a94d None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:22.380822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 141/277] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:22 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:22.391978 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed4914e-33f2-4c26-966f-22eead616ce2 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:22.392255 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed4914e-33f2-4c26-966f-22eead616ce2 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:22.392571 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed4914e-33f2-4c26-966f-22eead616ce2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:22.421901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-bed4914e-33f2-4c26-966f-22eead616ce2 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:22.423458 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-bed4914e-33f2-4c26-966f-22eead616ce2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:22.485403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a9d835-3275-474d-9603-5e803b35e422 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:22.537811 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 142/278] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:14:22.561407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7377bd73-4035-4495-9fcf-67c29bf2acbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:22.562468 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7377bd73-4035-4495-9fcf-67c29bf2acbc None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:22.562950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7377bd73-4035-4495-9fcf-67c29bf2acbc None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:22.563390 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7377bd73-4035-4495-9fcf-67c29bf2acbc None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:22.564570 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7377bd73-4035-4495-9fcf-67c29bf2acbc None admin] RBAC: Adding query filter params (name=member) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:22.565074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7377bd73-4035-4495-9fcf-67c29bf2acbc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:22.568090 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7377bd73-4035-4495-9fcf-67c29bf2acbc None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:22.585708 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 137/279] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 345 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:22.592655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed4914e-33f2-4c26-966f-22eead616ce2 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:22.637863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:22.773948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed4914e-33f2-4c26-966f-22eead616ce2 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:22.826206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04392580-675f-4098-b4d8-6a84d624abca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:22.827120 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04392580-675f-4098-b4d8-6a84d624abca None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:22.827550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04392580-675f-4098-b4d8-6a84d624abca None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:22.827973 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04392580-675f-4098-b4d8-6a84d624abca None admin] PATH_INFO: `/v3/users/f07dae83b6af43e4b2a9e1dcb594ab1b` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:22.853423 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04392580-675f-4098-b4d8-6a84d624abca None admin] RBAC: Authorizing `identity:get_user(user_id=f07dae83b6af43e4b2a9e1dcb594ab1b)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:22.856153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04392580-675f-4098-b4d8-6a84d624abca None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:22.859887 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 143/280] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:22 2022] GET /identity/v3/users/f07dae83b6af43e4b2a9e1dcb594ab1b => generated 293 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:14:22.869858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7377bd73-4035-4495-9fcf-67c29bf2acbc None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:23.078957 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:23.132209 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7377bd73-4035-4495-9fcf-67c29bf2acbc None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:23.133215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04392580-675f-4098-b4d8-6a84d624abca None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:23.245026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a720f807-68f5-42a9-b7d0-8dc3beec197b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:23.246113 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a720f807-68f5-42a9-b7d0-8dc3beec197b None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:23.246549 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a720f807-68f5-42a9-b7d0-8dc3beec197b None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:23.250024 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a720f807-68f5-42a9-b7d0-8dc3beec197b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:23.251622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a720f807-68f5-42a9-b7d0-8dc3beec197b None admin] RBAC: Adding query filter params (role.id=bc6aeaa495ca4f3dbe72905c744c2233, user.id=f07dae83b6af43e4b2a9e1dcb594ab1b, scope.system=all) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:23.252100 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a720f807-68f5-42a9-b7d0-8dc3beec197b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:23.255520 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a720f807-68f5-42a9-b7d0-8dc3beec197b None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:23.283508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 138/281] 10.222.0.64 () {56 vars in 1308 bytes} [Tue Oct 4 14:14:22 2022] GET /identity/v3/role_assignments?user.id=f07dae83b6af43e4b2a9e1dcb594ab1b&scope.system=all&role.id=bc6aeaa495ca4f3dbe72905c744c2233 => generated 226 bytes in 415 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:23.369080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:23.449948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04392580-675f-4098-b4d8-6a84d624abca None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:23.558188 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3acd00-769b-4061-8496-9dfd821a98c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:23.559278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3acd00-769b-4061-8496-9dfd821a98c1 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:23.559759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3acd00-769b-4061-8496-9dfd821a98c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:23.560245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3acd00-769b-4061-8496-9dfd821a98c1 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:23.590171 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3acd00-769b-4061-8496-9dfd821a98c1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:23.600826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-1a3acd00-769b-4061-8496-9dfd821a98c1 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:14:23.600826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:23.600826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:23.600826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:23.600826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:23.600826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:23.600826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:23.600826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:23.600826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:23.600826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:23.600826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:23.600826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:23.600826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:23.600826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:23.600826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:23.600826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:23.601826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:23.601826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:23.601826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:23.601826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:23.601826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:23.601826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:23.601826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:23.601826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:23.601826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:23.601826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:23.601826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:23.601826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:23.601826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:23.601826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:23.601826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:23.601826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:23.602629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:23.602629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:23.602629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:23.602629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:23.602629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:23.602629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:23.602629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:23.602629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:23.602629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:23.602629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:23.602629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:14:23.602629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:23.603276 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 144/282] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:23 2022] GET /identity/v3/roles/reader => generated 84 bytes in 470 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:23.612864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a720f807-68f5-42a9-b7d0-8dc3beec197b None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:23.808543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a720f807-68f5-42a9-b7d0-8dc3beec197b None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:23.864981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ee8c3d-2277-4220-b52e-06652f01c772 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:23.865610 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ee8c3d-2277-4220-b52e-06652f01c772 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:23.865891 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ee8c3d-2277-4220-b52e-06652f01c772 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:23.866164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ee8c3d-2277-4220-b52e-06652f01c772 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:23.866858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ee8c3d-2277-4220-b52e-06652f01c772 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:23.867160 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ee8c3d-2277-4220-b52e-06652f01c772 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:23.868620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ee8c3d-2277-4220-b52e-06652f01c772 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:23.886533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 139/283] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:23 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:23.896288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3acd00-769b-4061-8496-9dfd821a98c1 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:24.039115 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b789ba-e485-476a-b0c9-50615695570b None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:24.039674 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b789ba-e485-476a-b0c9-50615695570b None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:24.040166 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b789ba-e485-476a-b0c9-50615695570b None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:24.043051 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 140/284] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:24 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:14:24.051660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c107c1f6-cb6b-483b-a98f-dd8af6f23201 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:24.052136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c107c1f6-cb6b-483b-a98f-dd8af6f23201 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:24.052677 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c107c1f6-cb6b-483b-a98f-dd8af6f23201 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:24.073704 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3acd00-769b-4061-8496-9dfd821a98c1 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:24.100970 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-c107c1f6-cb6b-483b-a98f-dd8af6f23201 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:24.104349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c107c1f6-cb6b-483b-a98f-dd8af6f23201 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:24.108967 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:24.136238 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7eb1d9-b384-49cd-9880-59c456f82b31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:24.137748 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7eb1d9-b384-49cd-9880-59c456f82b31 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:24.138227 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7eb1d9-b384-49cd-9880-59c456f82b31 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:24.138719 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7eb1d9-b384-49cd-9880-59c456f82b31 None admin] PATH_INFO: `/v3/users/167ee884b89641bfa8a5cc73e6d7267d` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:24.140230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7eb1d9-b384-49cd-9880-59c456f82b31 None admin] RBAC: Authorizing `identity:get_user(user_id=167ee884b89641bfa8a5cc73e6d7267d)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:24.142189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7eb1d9-b384-49cd-9880-59c456f82b31 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:24.145042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 145/285] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:23 2022] GET /identity/v3/users/167ee884b89641bfa8a5cc73e6d7267d => generated 297 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:24.154944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7eb1d9-b384-49cd-9880-59c456f82b31 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:24.160784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:24.306081 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 141/286] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:24.428970 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7eb1d9-b384-49cd-9880-59c456f82b31 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:24.488354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae5f8f9-6431-44a6-be65-cbe50483b0e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:24.489365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae5f8f9-6431-44a6-be65-cbe50483b0e4 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:24.489858 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae5f8f9-6431-44a6-be65-cbe50483b0e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:24.490311 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae5f8f9-6431-44a6-be65-cbe50483b0e4 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:24.491808 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae5f8f9-6431-44a6-be65-cbe50483b0e4 None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:24.493538 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae5f8f9-6431-44a6-be65-cbe50483b0e4 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:24.495618 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 146/287] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:24 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 341 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:24.506177 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c107c1f6-cb6b-483b-a98f-dd8af6f23201 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:24.695162 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae5f8f9-6431-44a6-be65-cbe50483b0e4 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:24.788603 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c107c1f6-cb6b-483b-a98f-dd8af6f23201 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:24.890359 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b55fc4b-9126-4044-9c98-3dac95aa1488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:24.891068 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b55fc4b-9126-4044-9c98-3dac95aa1488 None admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:24.891462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b55fc4b-9126-4044-9c98-3dac95aa1488 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:24.891752 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b55fc4b-9126-4044-9c98-3dac95aa1488 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535/users/167ee884b89641bfa8a5cc73e6d7267d/roles/f425b98e3acf4e13b0da24b21e59cfce` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:24.897897 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b55fc4b-9126-4044-9c98-3dac95aa1488 None admin] RBAC: Authorizing `identity:create_grant(project_id=2728b0ff765c4e9fa2c0cb33c62b0535, user_id=167ee884b89641bfa8a5cc73e6d7267d, role_id=f425b98e3acf4e13b0da24b21e59cfce)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:24.899422 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b55fc4b-9126-4044-9c98-3dac95aa1488 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:24.907730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:24.927081 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:24.938920 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 142/288] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:14:24 2022] PUT /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535/users/167ee884b89641bfa8a5cc73e6d7267d/roles/f425b98e3acf4e13b0da24b21e59cfce => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:14:24.973108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae5f8f9-6431-44a6-be65-cbe50483b0e4 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:25.086805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9db47e-7679-4a27-9eab-0982f533e1af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:25.087945 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9db47e-7679-4a27-9eab-0982f533e1af None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:25.088459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9db47e-7679-4a27-9eab-0982f533e1af None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:25.088932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9db47e-7679-4a27-9eab-0982f533e1af None admin] PATH_INFO: `/v3/roles/member` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:25.122099 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9db47e-7679-4a27-9eab-0982f533e1af None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:25.132633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-ff9db47e-7679-4a27-9eab-0982f533e1af None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:25.132633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:25.132633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:25.132633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:25.132633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:25.132633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:25.132633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:25.132633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:25.132633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:25.132633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:25.132633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:25.132633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:25.132633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:25.132633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:25.132633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:25.132633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:25.134445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:25.134445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:25.134445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:25.134445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:25.134445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:25.134445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:25.134445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:25.134445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:25.134445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:25.134445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:25.134445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:25.134445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:25.134445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:25.134445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:25.134445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:25.134445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:25.136023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:25.136023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:25.136023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:25.136023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:25.136023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:25.136023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:25.136023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:25.136023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:25.136023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:25.136023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:25.136023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:25.136023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:25.136023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 147/289] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 440 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:14:25.141343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b55fc4b-9126-4044-9c98-3dac95aa1488 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:25.371253 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b55fc4b-9126-4044-9c98-3dac95aa1488 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:25.438934 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da69e0c-15a8-4818-956e-f379f9bf1713 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:25.439574 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da69e0c-15a8-4818-956e-f379f9bf1713 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:25.439861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da69e0c-15a8-4818-956e-f379f9bf1713 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:25.440132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da69e0c-15a8-4818-956e-f379f9bf1713 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:25.440844 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da69e0c-15a8-4818-956e-f379f9bf1713 None admin] RBAC: Adding query filter params (name=member) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:25.441187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da69e0c-15a8-4818-956e-f379f9bf1713 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:25.442851 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da69e0c-15a8-4818-956e-f379f9bf1713 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:25.459809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 143/290] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:25.466393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9db47e-7679-4a27-9eab-0982f533e1af None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:25.691547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9db47e-7679-4a27-9eab-0982f533e1af None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:25.740451 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:25.746771 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc517f45-bf5d-449a-b683-2c71661e2f5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:25.748136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc517f45-bf5d-449a-b683-2c71661e2f5c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:25.748917 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc517f45-bf5d-449a-b683-2c71661e2f5c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:25.749653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc517f45-bf5d-449a-b683-2c71661e2f5c None admin] PATH_INFO: `/v3/users/75cd754f0a2b4f4b83d337bdeb2f2c36` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:25.751025 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc517f45-bf5d-449a-b683-2c71661e2f5c None admin] RBAC: Authorizing `identity:get_user(user_id=75cd754f0a2b4f4b83d337bdeb2f2c36)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:25.753148 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc517f45-bf5d-449a-b683-2c71661e2f5c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:25.755652 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 148/291] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:25 2022] GET /identity/v3/users/75cd754f0a2b4f4b83d337bdeb2f2c36 => generated 297 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:25.762125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da69e0c-15a8-4818-956e-f379f9bf1713 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:25.872275 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:25.986208 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da69e0c-15a8-4818-956e-f379f9bf1713 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:26.062911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57e136b-1132-41f2-8ea3-f7c28511edc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:26.063969 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57e136b-1132-41f2-8ea3-f7c28511edc0 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:26.064464 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57e136b-1132-41f2-8ea3-f7c28511edc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:26.064944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57e136b-1132-41f2-8ea3-f7c28511edc0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:26.188595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0494cb5-9fdd-4b26-9821-bc9341276c19 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:26.190139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0494cb5-9fdd-4b26-9821-bc9341276c19 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:26.190925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0494cb5-9fdd-4b26-9821-bc9341276c19 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:26.200348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 149/292] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:26 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:26.205313 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e4bc0e-49b8-41b8-a7a8-7c5fd3d34ed8 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:26.205795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e4bc0e-49b8-41b8-a7a8-7c5fd3d34ed8 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:26.206268 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e4bc0e-49b8-41b8-a7a8-7c5fd3d34ed8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:26.230223 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57e136b-1132-41f2-8ea3-f7c28511edc0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:26.232123 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57e136b-1132-41f2-8ea3-f7c28511edc0 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:26.252610 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-c1e4bc0e-49b8-41b8-a7a8-7c5fd3d34ed8 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:26.254246 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c1e4bc0e-49b8-41b8-a7a8-7c5fd3d34ed8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:26.349214 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:26.376444 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 150/293] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:14:26.381677 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74927c3d-ae41-4d82-b393-3ee37c4d9f29 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:26.384587 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74927c3d-ae41-4d82-b393-3ee37c4d9f29 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:26.388195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74927c3d-ae41-4d82-b393-3ee37c4d9f29 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:26.393912 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 151/294] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:26 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:26.400539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0225ced1-6645-4863-9ef9-a1ed7b7ad98e None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:26.401460 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0225ced1-6645-4863-9ef9-a1ed7b7ad98e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:26.402211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0225ced1-6645-4863-9ef9-a1ed7b7ad98e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:26.436251 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-0225ced1-6645-4863-9ef9-a1ed7b7ad98e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:26.438040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-0225ced1-6645-4863-9ef9-a1ed7b7ad98e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:26.501781 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 144/295] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 740 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 14:14:26.508132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57e136b-1132-41f2-8ea3-f7c28511edc0 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:26.599249 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 152/296] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:26.736149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57e136b-1132-41f2-8ea3-f7c28511edc0 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:26.804574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da9ab29-b2a7-47b9-952c-2da40c2c2b18 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:26.804879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da9ab29-b2a7-47b9-952c-2da40c2c2b18 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:26.805334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da9ab29-b2a7-47b9-952c-2da40c2c2b18 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:26.808757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 153/297] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:26.813673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cca537b-ece7-48fb-afb7-bacc07f9e89b None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:26.814876 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cca537b-ece7-48fb-afb7-bacc07f9e89b None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:26.815371 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cca537b-ece7-48fb-afb7-bacc07f9e89b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:26.836056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5588df85-4434-4b8f-8b9c-df2b01cf8f33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:26.836758 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5588df85-4434-4b8f-8b9c-df2b01cf8f33 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:26.837041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5588df85-4434-4b8f-8b9c-df2b01cf8f33 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:26.837318 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5588df85-4434-4b8f-8b9c-df2b01cf8f33 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:26.838766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5588df85-4434-4b8f-8b9c-df2b01cf8f33 None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:26.840449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5588df85-4434-4b8f-8b9c-df2b01cf8f33 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:26.842532 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 145/298] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:26 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 335 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:14:26.849044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5588df85-4434-4b8f-8b9c-df2b01cf8f33 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:26.852540 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-1cca537b-ece7-48fb-afb7-bacc07f9e89b None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:26.854748 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1cca537b-ece7-48fb-afb7-bacc07f9e89b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:26.892461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:26.946937 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:26.958554 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 154/299] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:26.959592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cca537b-ece7-48fb-afb7-bacc07f9e89b None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:27.022920 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5588df85-4434-4b8f-8b9c-df2b01cf8f33 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:27.092087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cca537b-ece7-48fb-afb7-bacc07f9e89b None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:27.117139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b80e53-1268-4d83-8ebc-c332db716d0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:27.117814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b80e53-1268-4d83-8ebc-c332db716d0f None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:27.118099 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b80e53-1268-4d83-8ebc-c332db716d0f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:27.118376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b80e53-1268-4d83-8ebc-c332db716d0f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:27.119680 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b80e53-1268-4d83-8ebc-c332db716d0f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:27.121553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b80e53-1268-4d83-8ebc-c332db716d0f None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:27.122924 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 146/300] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:14:26 2022] GET /identity/v3/domains/default => generated 199 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:27.133192 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b80e53-1268-4d83-8ebc-c332db716d0f None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:27.135237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a766fa2-b482-45d0-8a29-a9c5233cac91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:27.136121 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a766fa2-b482-45d0-8a29-a9c5233cac91 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:27.136613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a766fa2-b482-45d0-8a29-a9c5233cac91 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:27.137063 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a766fa2-b482-45d0-8a29-a9c5233cac91 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:27.138041 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a766fa2-b482-45d0-8a29-a9c5233cac91 None admin] RBAC: Adding query filter params (role.id=bc6aeaa495ca4f3dbe72905c744c2233, scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535, user.id=75cd754f0a2b4f4b83d337bdeb2f2c36) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:27.138549 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a766fa2-b482-45d0-8a29-a9c5233cac91 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:27.140550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a766fa2-b482-45d0-8a29-a9c5233cac91 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:27.148453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 155/301] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:14:26 2022] GET /identity/v3/role_assignments?user.id=75cd754f0a2b4f4b83d337bdeb2f2c36&scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535&role.id=bc6aeaa495ca4f3dbe72905c744c2233 => generated 608 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:27.194505 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a766fa2-b482-45d0-8a29-a9c5233cac91 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:27.357013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:27.376513 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:27.392678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b80e53-1268-4d83-8ebc-c332db716d0f None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:27.441707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef5fb74-8728-477d-911b-3458af84dfa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:27.442346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef5fb74-8728-477d-911b-3458af84dfa9 None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:27.442632 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef5fb74-8728-477d-911b-3458af84dfa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:27.442910 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef5fb74-8728-477d-911b-3458af84dfa9 None admin] PATH_INFO: `/v3/groups` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:27.443565 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef5fb74-8728-477d-911b-3458af84dfa9 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:27.445165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef5fb74-8728-477d-911b-3458af84dfa9 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:27.450700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a766fa2-b482-45d0-8a29-a9c5233cac91 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:27.476256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 147/302] 10.222.0.64 () {60 vars in 1134 bytes} [Tue Oct 4 14:14:27 2022] POST /identity/v3/groups => generated 229 bytes in 343 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:14:27.477193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef5fb74-8728-477d-911b-3458af84dfa9 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:27.518446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f56847-c9d2-4043-93ef-64416fb4d2b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:27.520806 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f56847-c9d2-4043-93ef-64416fb4d2b2 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:27.521383 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f56847-c9d2-4043-93ef-64416fb4d2b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:27.521903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f56847-c9d2-4043-93ef-64416fb4d2b2 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:27.537473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f56847-c9d2-4043-93ef-64416fb4d2b2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:27.545071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-52f56847-c9d2-4043-93ef-64416fb4d2b2 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:14:27.545071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:27.545071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:27.545071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:27.545071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:27.545071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:27.545071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:27.545071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:27.545071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:27.545071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:27.545071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:27.545071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:27.545071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:27.545071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:27.545071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:27.545071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:27.546065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:27.546065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:27.546065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:27.546065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:27.546065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:27.546065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:27.546065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:27.546065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:27.546065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:27.546065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:27.546065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:27.546065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:27.546065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:27.546065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:27.546065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:27.546065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:27.546879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:27.546879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:27.546879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:27.546879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:27.546879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:27.546879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:27.546879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:27.546879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:27.546879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:27.546879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:27.546879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:14:27.546879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:27.549699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 156/303] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:27 2022] GET /identity/v3/roles/reader => generated 84 bytes in 356 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:14:27.560142 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f56847-c9d2-4043-93ef-64416fb4d2b2 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:27.705245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f56847-c9d2-4043-93ef-64416fb4d2b2 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:27.713854 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef5fb74-8728-477d-911b-3458af84dfa9 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:27.728891 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:27.791409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce041490-d006-47e5-a715-b201ff1ad6d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:27.792449 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce041490-d006-47e5-a715-b201ff1ad6d7 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:27.792941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce041490-d006-47e5-a715-b201ff1ad6d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:27.793416 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce041490-d006-47e5-a715-b201ff1ad6d7 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:27.794670 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce041490-d006-47e5-a715-b201ff1ad6d7 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:27.795152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce041490-d006-47e5-a715-b201ff1ad6d7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:27.797860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce041490-d006-47e5-a715-b201ff1ad6d7 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:27.813773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11950f1-6711-447f-bc33-0f21bc0cbfa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:27.814832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11950f1-6711-447f-bc33-0f21bc0cbfa5 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:27.815332 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11950f1-6711-447f-bc33-0f21bc0cbfa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:27.815757 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11950f1-6711-447f-bc33-0f21bc0cbfa5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:27.818252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 157/304] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:27 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:27.829850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce041490-d006-47e5-a715-b201ff1ad6d7 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:27.849619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11950f1-6711-447f-bc33-0f21bc0cbfa5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:27.855694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-e11950f1-6711-447f-bc33-0f21bc0cbfa5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:27.855694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:27.855694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:27.855694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:27.855694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:27.855694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:27.855694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:27.855694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:27.855694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:27.855694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:27.855694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:27.855694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:27.855694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:27.855694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:27.855694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:27.855694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:27.857713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:27.857713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:27.857713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:27.857713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:27.857713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:27.857713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:27.857713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:27.857713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:27.857713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:27.857713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:27.857713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:27.857713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:27.857713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:27.857713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:27.857713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:27.857713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:27.859742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:27.859742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:27.859742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:27.859742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:27.859742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:27.859742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:27.859742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:27.859742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:27.859742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:27.859742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:27.859742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:27.859742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:27.859742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 148/305] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:27 2022] GET /identity/v3/roles/member => generated 84 bytes in 380 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:27.871291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11950f1-6711-447f-bc33-0f21bc0cbfa5 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:27.905692 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:28.035870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce041490-d006-47e5-a715-b201ff1ad6d7 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:28.062419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11950f1-6711-447f-bc33-0f21bc0cbfa5 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:28.100723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4df59a8-a00b-4509-96cf-27cd8ffc18e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:28.101537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4df59a8-a00b-4509-96cf-27cd8ffc18e2 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:28.101917 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4df59a8-a00b-4509-96cf-27cd8ffc18e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:28.102294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4df59a8-a00b-4509-96cf-27cd8ffc18e2 None admin] PATH_INFO: `/v3/users/3783179711a2422fb709fa511638266e` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:28.128421 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371d11a6-0f25-4ad2-a4be-58c22284083b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:28.128714 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4df59a8-a00b-4509-96cf-27cd8ffc18e2 None admin] RBAC: Authorizing `identity:get_user(user_id=3783179711a2422fb709fa511638266e)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:28.129345 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371d11a6-0f25-4ad2-a4be-58c22284083b None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:28.129728 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371d11a6-0f25-4ad2-a4be-58c22284083b None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:28.130107 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371d11a6-0f25-4ad2-a4be-58c22284083b None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:28.130908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371d11a6-0f25-4ad2-a4be-58c22284083b None admin] RBAC: Adding query filter params (name=member) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:28.131071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4df59a8-a00b-4509-96cf-27cd8ffc18e2 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:28.131400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371d11a6-0f25-4ad2-a4be-58c22284083b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:28.132943 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371d11a6-0f25-4ad2-a4be-58c22284083b None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:28.134020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 158/306] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:27 2022] GET /identity/v3/users/3783179711a2422fb709fa511638266e => generated 293 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:28.143443 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4df59a8-a00b-4509-96cf-27cd8ffc18e2 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:28.146215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 149/307] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:14:28.157781 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371d11a6-0f25-4ad2-a4be-58c22284083b None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:28.224679 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:28.341525 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:28.395572 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371d11a6-0f25-4ad2-a4be-58c22284083b None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:28.398036 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4df59a8-a00b-4509-96cf-27cd8ffc18e2 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:28.447432 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f428ad35-cc01-4fcb-a456-7c7dc21c4b05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:28.448495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f428ad35-cc01-4fcb-a456-7c7dc21c4b05 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:28.448806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f428ad35-cc01-4fcb-a456-7c7dc21c4b05 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:28.449048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f428ad35-cc01-4fcb-a456-7c7dc21c4b05 None admin] PATH_INFO: `/v3/users/f07dae83b6af43e4b2a9e1dcb594ab1b` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:28.452087 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f428ad35-cc01-4fcb-a456-7c7dc21c4b05 None admin] RBAC: Authorizing `identity:get_user(user_id=f07dae83b6af43e4b2a9e1dcb594ab1b)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:28.453663 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f428ad35-cc01-4fcb-a456-7c7dc21c4b05 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:28.455390 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 150/308] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:28 2022] GET /identity/v3/users/f07dae83b6af43e4b2a9e1dcb594ab1b => generated 293 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:28.456838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20c8934-ba60-4494-a9ab-1c128da2ff41 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:28.457297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20c8934-ba60-4494-a9ab-1c128da2ff41 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:28.457649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20c8934-ba60-4494-a9ab-1c128da2ff41 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:28.459423 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 151/309] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:28.465016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20c8934-ba60-4494-a9ab-1c128da2ff41 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:28.482438 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2e3103-20f3-4283-920e-8264d99eca87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:28.483856 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2e3103-20f3-4283-920e-8264d99eca87 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:28.484391 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2e3103-20f3-4283-920e-8264d99eca87 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:28.484866 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2e3103-20f3-4283-920e-8264d99eca87 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:28.486236 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2e3103-20f3-4283-920e-8264d99eca87 None admin] RBAC: Adding query filter params (role.id=f425b98e3acf4e13b0da24b21e59cfce, user.id=3783179711a2422fb709fa511638266e, scope.system=all) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:28.486785 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2e3103-20f3-4283-920e-8264d99eca87 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:28.489259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2e3103-20f3-4283-920e-8264d99eca87 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:28.498120 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 159/310] 10.222.0.64 () {56 vars in 1308 bytes} [Tue Oct 4 14:14:28 2022] GET /identity/v3/role_assignments?user.id=3783179711a2422fb709fa511638266e&scope.system=all&role.id=f425b98e3acf4e13b0da24b21e59cfce => generated 226 bytes in 356 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:28.501870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ea6e2-55bd-4ec4-aa44-5938c99e90ae None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:28.502279 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ea6e2-55bd-4ec4-aa44-5938c99e90ae None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:28.502442 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ea6e2-55bd-4ec4-aa44-5938c99e90ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:28.524238 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-9f9ea6e2-55bd-4ec4-aa44-5938c99e90ae None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:28.525709 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9f9ea6e2-55bd-4ec4-aa44-5938c99e90ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:28.618656 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 160/311] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:28.631569 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20c8934-ba60-4494-a9ab-1c128da2ff41 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:28.686217 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b88c166-62d8-4a1c-abcb-5403fa938536 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:28.687313 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b88c166-62d8-4a1c-abcb-5403fa938536 None admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:28.687848 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b88c166-62d8-4a1c-abcb-5403fa938536 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:28.688358 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b88c166-62d8-4a1c-abcb-5403fa938536 None admin] PATH_INFO: `/v3/system/users/f07dae83b6af43e4b2a9e1dcb594ab1b/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:28.690302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b88c166-62d8-4a1c-abcb-5403fa938536 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=f07dae83b6af43e4b2a9e1dcb594ab1b, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:28.692664 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b88c166-62d8-4a1c-abcb-5403fa938536 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:28.705836 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 152/312] 10.222.0.64 () {58 vars in 1334 bytes} [Tue Oct 4 14:14:28 2022] PUT /identity/v3/system/users/f07dae83b6af43e4b2a9e1dcb594ab1b/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:14:28.909002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9ea6e2-55bd-4ec4-aa44-5938c99e90ae None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:28.950965 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:29.038864 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9ea6e2-55bd-4ec4-aa44-5938c99e90ae None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:29.080644 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033a8833-9caf-45ac-935a-6979fec9411b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:29.081345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033a8833-9caf-45ac-935a-6979fec9411b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:29.081753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033a8833-9caf-45ac-935a-6979fec9411b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:29.082229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033a8833-9caf-45ac-935a-6979fec9411b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:29.095094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033a8833-9caf-45ac-935a-6979fec9411b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:29.100730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-033a8833-9caf-45ac-935a-6979fec9411b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:14:29.100730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:29.100730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:29.100730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:29.100730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:29.100730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:29.100730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:29.100730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:29.100730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:29.100730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:29.100730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:29.100730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:29.100730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:29.100730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:29.100730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:29.100730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:29.101708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:29.101708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:29.101708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:29.101708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:29.101708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:29.101708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:29.101708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:29.101708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:29.101708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:29.101708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:29.101708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:29.101708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:29.101708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:29.101708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:29.101708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:29.101708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:29.102516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:29.102516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:29.102516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:29.102516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:29.102516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:29.102516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:29.102516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:29.102516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:29.102516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:29.102516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:29.102516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:14:29.102516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:29.102516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 161/313] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:28 2022] GET /identity/v3/roles/reader => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:29.109902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b88c166-62d8-4a1c-abcb-5403fa938536 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:29.126341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:29.253038 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b88c166-62d8-4a1c-abcb-5403fa938536 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:29.296818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d50af3d-0e15-401e-991c-8c332a976d6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:29.297901 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d50af3d-0e15-401e-991c-8c332a976d6a None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:29.298425 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d50af3d-0e15-401e-991c-8c332a976d6a None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:29.298948 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d50af3d-0e15-401e-991c-8c332a976d6a None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:29.299933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d50af3d-0e15-401e-991c-8c332a976d6a None admin] RBAC: Adding query filter params (name=reader) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:29.301103 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d50af3d-0e15-401e-991c-8c332a976d6a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:29.302774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d50af3d-0e15-401e-991c-8c332a976d6a None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:29.313272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 153/314] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:29 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:29.320150 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033a8833-9caf-45ac-935a-6979fec9411b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:29.433036 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:29.449979 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033a8833-9caf-45ac-935a-6979fec9411b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:29.488925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73cd4d0-4573-42df-8bfc-1048e5e47b81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:29.489630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73cd4d0-4573-42df-8bfc-1048e5e47b81 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:29.489911 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73cd4d0-4573-42df-8bfc-1048e5e47b81 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:29.490182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73cd4d0-4573-42df-8bfc-1048e5e47b81 None admin] PATH_INFO: `/v3/users/167ee884b89641bfa8a5cc73e6d7267d` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:29.491557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73cd4d0-4573-42df-8bfc-1048e5e47b81 None admin] RBAC: Authorizing `identity:get_user(user_id=167ee884b89641bfa8a5cc73e6d7267d)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:29.493103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73cd4d0-4573-42df-8bfc-1048e5e47b81 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:29.495074 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 162/315] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:29 2022] GET /identity/v3/users/167ee884b89641bfa8a5cc73e6d7267d => generated 297 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:29.501235 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d50af3d-0e15-401e-991c-8c332a976d6a None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:29.595210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:29.635637 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d50af3d-0e15-401e-991c-8c332a976d6a None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:29.677473 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a94e4e1-08fc-4f0c-abe0-3f8b9bad5960 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:29.678154 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a94e4e1-08fc-4f0c-abe0-3f8b9bad5960 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:29.678444 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a94e4e1-08fc-4f0c-abe0-3f8b9bad5960 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:29.678732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a94e4e1-08fc-4f0c-abe0-3f8b9bad5960 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:29.762930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a94e4e1-08fc-4f0c-abe0-3f8b9bad5960 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:29.764474 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a94e4e1-08fc-4f0c-abe0-3f8b9bad5960 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:29.850611 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:29.888203 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 154/316] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:14:29.900798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73cd4d0-4573-42df-8bfc-1048e5e47b81 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:30.106088 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73cd4d0-4573-42df-8bfc-1048e5e47b81 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:30.177423 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcda4b4-da6a-4256-8525-4cd64cfd682a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:30.179000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcda4b4-da6a-4256-8525-4cd64cfd682a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:30.179760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcda4b4-da6a-4256-8525-4cd64cfd682a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:30.180527 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcda4b4-da6a-4256-8525-4cd64cfd682a None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:30.183049 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcda4b4-da6a-4256-8525-4cd64cfd682a None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:30.186226 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcda4b4-da6a-4256-8525-4cd64cfd682a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:30.190324 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 163/317] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:29 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:30.201117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a94e4e1-08fc-4f0c-abe0-3f8b9bad5960 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:30.329213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a94e4e1-08fc-4f0c-abe0-3f8b9bad5960 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:30.373458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8b82ca-147c-4cd8-ac75-be01cab01809 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:30.374143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8b82ca-147c-4cd8-ac75-be01cab01809 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:30.374480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8b82ca-147c-4cd8-ac75-be01cab01809 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:30.374761 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8b82ca-147c-4cd8-ac75-be01cab01809 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:30.375537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8b82ca-147c-4cd8-ac75-be01cab01809 None admin] RBAC: Adding query filter params (role.id=f425b98e3acf4e13b0da24b21e59cfce, scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535, user.id=167ee884b89641bfa8a5cc73e6d7267d) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:30.375854 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8b82ca-147c-4cd8-ac75-be01cab01809 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:30.377472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8b82ca-147c-4cd8-ac75-be01cab01809 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:30.384972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 155/318] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:14:30 2022] GET /identity/v3/role_assignments?user.id=167ee884b89641bfa8a5cc73e6d7267d&scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535&role.id=f425b98e3acf4e13b0da24b21e59cfce => generated 608 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:31.106867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507480bf-057e-414f-93d2-1f8b4861c6ce None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:31.107590 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507480bf-057e-414f-93d2-1f8b4861c6ce None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:31.107993 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507480bf-057e-414f-93d2-1f8b4861c6ce None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:31.110402 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 164/319] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:31 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:31.120089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc45e52-51d5-465b-82a3-6a769a2aedac None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:31.120624 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc45e52-51d5-465b-82a3-6a769a2aedac None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:31.121113 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc45e52-51d5-465b-82a3-6a769a2aedac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:31.152491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-5bc45e52-51d5-465b-82a3-6a769a2aedac None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:31.154607 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-5bc45e52-51d5-465b-82a3-6a769a2aedac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:31.212207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:31.288462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 156/320] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:31.571898 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507480bf-057e-414f-93d2-1f8b4861c6ce None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:31.605091 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:31.613847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f370d89-a8b1-4ed4-9fa3-cba4b94d42b1 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:31.615191 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f370d89-a8b1-4ed4-9fa3-cba4b94d42b1 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:31.616471 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f370d89-a8b1-4ed4-9fa3-cba4b94d42b1 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:31.619222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 157/321] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:31 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:31.628963 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0f23f5-c0ae-42b9-9ae9-377b47be6495 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:31.629411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0f23f5-c0ae-42b9-9ae9-377b47be6495 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:31.629706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0f23f5-c0ae-42b9-9ae9-377b47be6495 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:31.650732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-6c0f23f5-c0ae-42b9-9ae9-377b47be6495 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:31.652000 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-6c0f23f5-c0ae-42b9-9ae9-377b47be6495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:31.739448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507480bf-057e-414f-93d2-1f8b4861c6ce None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:31.750346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 158/322] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:14:31.752534 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9250c03-421f-425d-8fae-68dccd8bbe30 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:31.753019 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9250c03-421f-425d-8fae-68dccd8bbe30 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:31.753576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9250c03-421f-425d-8fae-68dccd8bbe30 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:31.756348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 159/323] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:31 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:31.764647 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d6049b-ed71-4add-b398-ff6e9c0f6b0d None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:31.765131 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d6049b-ed71-4add-b398-ff6e9c0f6b0d None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:31.765604 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d6049b-ed71-4add-b398-ff6e9c0f6b0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:31.793937 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-f7d6049b-ed71-4add-b398-ff6e9c0f6b0d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:31.795171 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-f7d6049b-ed71-4add-b398-ff6e9c0f6b0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:31.807677 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afe5531-41e3-4613-abe7-f463ccfedfb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:31.808988 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afe5531-41e3-4613-abe7-f463ccfedfb1 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:31.809277 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afe5531-41e3-4613-abe7-f463ccfedfb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:31.809568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afe5531-41e3-4613-abe7-f463ccfedfb1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:31.810789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afe5531-41e3-4613-abe7-f463ccfedfb1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:31.812258 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afe5531-41e3-4613-abe7-f463ccfedfb1 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:31.814211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 165/324] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:14:31 2022] GET /identity/v3/domains/default => generated 199 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:31.820007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afe5531-41e3-4613-abe7-f463ccfedfb1 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:31.908346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 160/325] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:31.963952 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:31.977090 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afe5531-41e3-4613-abe7-f463ccfedfb1 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:32.021439 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64d302b-5507-4f46-a6b4-6de4d8372003 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:32.021976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64d302b-5507-4f46-a6b4-6de4d8372003 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:32.022245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64d302b-5507-4f46-a6b4-6de4d8372003 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:32.022519 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64d302b-5507-4f46-a6b4-6de4d8372003 None admin] PATH_INFO: `/v3/groups` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:32.023183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64d302b-5507-4f46-a6b4-6de4d8372003 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:32.024697 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64d302b-5507-4f46-a6b4-6de4d8372003 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:32.050829 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 166/326] 10.222.0.64 () {60 vars in 1134 bytes} [Tue Oct 4 14:14:31 2022] POST /identity/v3/groups => generated 226 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:14:32.200202 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d6049b-ed71-4add-b398-ff6e9c0f6b0d None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:32.257370 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:32.334874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64d302b-5507-4f46-a6b4-6de4d8372003 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:32.379761 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d6049b-ed71-4add-b398-ff6e9c0f6b0d None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:32.457762 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ead1c51-0b2e-468c-9725-675e119a5c14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:32.458816 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ead1c51-0b2e-468c-9725-675e119a5c14 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:32.459282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ead1c51-0b2e-468c-9725-675e119a5c14 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:32.459737 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ead1c51-0b2e-468c-9725-675e119a5c14 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:32.480178 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ead1c51-0b2e-468c-9725-675e119a5c14 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:32.490048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-3ead1c51-0b2e-468c-9725-675e119a5c14 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:32.490048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:32.490048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:32.490048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:32.490048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:32.490048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:32.490048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:32.490048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:32.490048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:32.490048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:32.490048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:32.490048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:32.490048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:32.490048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:32.490048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:32.490048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:32.491033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:32.491033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:32.491033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:32.491033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:32.491033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:32.491033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:32.491033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:32.491033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:32.491033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:32.491033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:32.491033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:32.491033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:32.491033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:32.491033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:32.491033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:32.491033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:32.491840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:32.491840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:32.491840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:32.491840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:32.491840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:32.491840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:32.491840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:32.491840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:32.491840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:32.491840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:32.491840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:32.491840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:32.492520 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 161/327] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:32 2022] GET /identity/v3/roles/member => generated 84 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:32.500927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ead1c51-0b2e-468c-9725-675e119a5c14 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:32.546722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64d302b-5507-4f46-a6b4-6de4d8372003 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:32.623240 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328a7b7f-bd52-4647-870f-c10b49e1a719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:32.626487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328a7b7f-bd52-4647-870f-c10b49e1a719 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:32.626487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328a7b7f-bd52-4647-870f-c10b49e1a719 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:32.626487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328a7b7f-bd52-4647-870f-c10b49e1a719 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:32.653371 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328a7b7f-bd52-4647-870f-c10b49e1a719 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:32.666046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-328a7b7f-bd52-4647-870f-c10b49e1a719 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:14:32.666046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:32.666046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:32.666046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:32.666046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:32.666046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:32.666046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:32.666046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:32.666046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:32.666046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:32.666046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:32.666046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:32.666046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:32.666046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:32.666046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:32.666046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:32.667825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:32.667825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:32.667825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:32.667825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:32.667825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:32.667825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:32.667825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:32.667825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:32.667825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:32.667825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:32.667825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:32.667825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:32.667825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:32.667825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:32.667825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:32.667825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:32.669296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:32.669296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:32.669296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:32.669296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:32.669296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:32.669296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:32.669296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:32.669296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:32.669296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:32.669296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:32.669296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:14:32.669296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:32.669296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 167/328] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:32 2022] GET /identity/v3/roles/reader => generated 84 bytes in 335 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:32.678339 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328a7b7f-bd52-4647-870f-c10b49e1a719 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:32.710576 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:32.711749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:32.748725 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ead1c51-0b2e-468c-9725-675e119a5c14 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:32.839015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57b0b9e-78f5-4e2b-84d3-5c1047880585 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:32.840542 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57b0b9e-78f5-4e2b-84d3-5c1047880585 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:32.841285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57b0b9e-78f5-4e2b-84d3-5c1047880585 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:32.842008 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57b0b9e-78f5-4e2b-84d3-5c1047880585 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:32.843432 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57b0b9e-78f5-4e2b-84d3-5c1047880585 None admin] RBAC: Adding query filter params (name=member) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:32.844260 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57b0b9e-78f5-4e2b-84d3-5c1047880585 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:32.847017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57b0b9e-78f5-4e2b-84d3-5c1047880585 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:32.866617 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 162/329] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:32 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 367 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:14:32.878100 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57b0b9e-78f5-4e2b-84d3-5c1047880585 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:32.908807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328a7b7f-bd52-4647-870f-c10b49e1a719 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:32.976821 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2865af73-25d1-4851-9030-f202ab056747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:32.977783 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2865af73-25d1-4851-9030-f202ab056747 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:32.978237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2865af73-25d1-4851-9030-f202ab056747 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:32.978688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2865af73-25d1-4851-9030-f202ab056747 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:32.979824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2865af73-25d1-4851-9030-f202ab056747 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:32.980342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2865af73-25d1-4851-9030-f202ab056747 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:32.982837 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2865af73-25d1-4851-9030-f202ab056747 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:33.001580 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 168/330] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:32 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:14:33.013574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2865af73-25d1-4851-9030-f202ab056747 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:33.105276 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57b0b9e-78f5-4e2b-84d3-5c1047880585 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:33.185695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01ea318-b281-4704-b1ff-2390d72a6ffc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:33.186664 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01ea318-b281-4704-b1ff-2390d72a6ffc None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:33.187173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01ea318-b281-4704-b1ff-2390d72a6ffc None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:33.187736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01ea318-b281-4704-b1ff-2390d72a6ffc None admin] PATH_INFO: `/v3/users/f07dae83b6af43e4b2a9e1dcb594ab1b` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:33.190014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01ea318-b281-4704-b1ff-2390d72a6ffc None admin] RBAC: Authorizing `identity:get_user(user_id=f07dae83b6af43e4b2a9e1dcb594ab1b)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:33.193361 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01ea318-b281-4704-b1ff-2390d72a6ffc None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:33.196746 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 163/331] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:32 2022] GET /identity/v3/users/f07dae83b6af43e4b2a9e1dcb594ab1b => generated 293 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:33.200119 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:33.212419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01ea318-b281-4704-b1ff-2390d72a6ffc None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:33.257022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2865af73-25d1-4851-9030-f202ab056747 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:33.288388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:33.333170 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276c41d9-f450-48f0-8b0f-0dd4c08dfb73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:33.334218 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276c41d9-f450-48f0-8b0f-0dd4c08dfb73 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:33.334656 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276c41d9-f450-48f0-8b0f-0dd4c08dfb73 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:33.335323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276c41d9-f450-48f0-8b0f-0dd4c08dfb73 None admin] PATH_INFO: `/v3/users/3783179711a2422fb709fa511638266e` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:33.338647 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276c41d9-f450-48f0-8b0f-0dd4c08dfb73 None admin] RBAC: Authorizing `identity:get_user(user_id=3783179711a2422fb709fa511638266e)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:33.341397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276c41d9-f450-48f0-8b0f-0dd4c08dfb73 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:33.344810 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 169/332] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:33 2022] GET /identity/v3/users/3783179711a2422fb709fa511638266e => generated 293 bytes in 332 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:33.353497 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276c41d9-f450-48f0-8b0f-0dd4c08dfb73 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:33.429828 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01ea318-b281-4704-b1ff-2390d72a6ffc None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:33.560301 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc551e4-d2c8-498c-84db-6166f5ebdbef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:33.561391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc551e4-d2c8-498c-84db-6166f5ebdbef None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:33.561882 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc551e4-d2c8-498c-84db-6166f5ebdbef None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:33.562374 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc551e4-d2c8-498c-84db-6166f5ebdbef None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:33.564091 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc551e4-d2c8-498c-84db-6166f5ebdbef None admin] RBAC: Adding query filter params (role.id=bc6aeaa495ca4f3dbe72905c744c2233, user.id=f07dae83b6af43e4b2a9e1dcb594ab1b, scope.system=all) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:33.564689 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc551e4-d2c8-498c-84db-6166f5ebdbef None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:33.568151 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc551e4-d2c8-498c-84db-6166f5ebdbef None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:33.628876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 164/333] 10.222.0.64 () {56 vars in 1308 bytes} [Tue Oct 4 14:14:33 2022] GET /identity/v3/role_assignments?user.id=f07dae83b6af43e4b2a9e1dcb594ab1b&scope.system=all&role.id=bc6aeaa495ca4f3dbe72905c744c2233 => generated 510 bytes in 417 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:14:33.656787 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276c41d9-f450-48f0-8b0f-0dd4c08dfb73 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:33.758690 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e19c574-27da-428d-aca5-ea06769caf73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:33.759678 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e19c574-27da-428d-aca5-ea06769caf73 None admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:33.760101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e19c574-27da-428d-aca5-ea06769caf73 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:33.760620 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e19c574-27da-428d-aca5-ea06769caf73 None admin] PATH_INFO: `/v3/system/users/3783179711a2422fb709fa511638266e/roles/f425b98e3acf4e13b0da24b21e59cfce` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:33.765140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e19c574-27da-428d-aca5-ea06769caf73 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=3783179711a2422fb709fa511638266e, role_id=f425b98e3acf4e13b0da24b21e59cfce)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:33.768149 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e19c574-27da-428d-aca5-ea06769caf73 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:33.789691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 170/334] 10.222.0.64 () {58 vars in 1334 bytes} [Tue Oct 4 14:14:33 2022] PUT /identity/v3/system/users/3783179711a2422fb709fa511638266e/roles/f425b98e3acf4e13b0da24b21e59cfce => generated 0 bytes in 437 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:14:37.189265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eab0737-cf3e-48cb-ba63-5add6b91b1cd None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:37.191428 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eab0737-cf3e-48cb-ba63-5add6b91b1cd None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:37.192249 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eab0737-cf3e-48cb-ba63-5add6b91b1cd None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:37.194496 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 165/335] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:37 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:37.203801 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd9775c-053a-4ebf-9697-88399fa8c98d None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:37.204362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd9775c-053a-4ebf-9697-88399fa8c98d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:37.204799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd9775c-053a-4ebf-9697-88399fa8c98d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:37.230839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:37.249209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-2dd9775c-053a-4ebf-9697-88399fa8c98d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:37.255543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-2dd9775c-053a-4ebf-9697-88399fa8c98d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:37.425342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 171/336] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:14:37.857215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea551607-638c-4569-af98-d290e512da89 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:37.857927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea551607-638c-4569-af98-d290e512da89 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:37.858420 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea551607-638c-4569-af98-d290e512da89 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:37.859507 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c097343c-44f9-49a9-9881-daf9ded8e830 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:37.859793 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c097343c-44f9-49a9-9881-daf9ded8e830 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:37.860069 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c097343c-44f9-49a9-9881-daf9ded8e830 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:37.861033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 166/337] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:37.861776 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 172/338] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:37.867656 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95016e19-9c2f-4eaf-b643-8d5a4c1bb6b4 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:37.868286 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95016e19-9c2f-4eaf-b643-8d5a4c1bb6b4 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:37.868831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95016e19-9c2f-4eaf-b643-8d5a4c1bb6b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:37.873792 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ad9031-1f9e-4ec4-9686-a1b4e479b0e6 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:37.874452 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ad9031-1f9e-4ec4-9686-a1b4e479b0e6 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:37.874944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ad9031-1f9e-4ec4-9686-a1b4e479b0e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:37.910568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-86ad9031-1f9e-4ec4-9686-a1b4e479b0e6 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:37.910793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-95016e19-9c2f-4eaf-b643-8d5a4c1bb6b4 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:37.912287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-86ad9031-1f9e-4ec4-9686-a1b4e479b0e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:37.912817 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-95016e19-9c2f-4eaf-b643-8d5a4c1bb6b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:38.035666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 173/339] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:38.089656 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 167/340] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:38.338241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ad9031-1f9e-4ec4-9686-a1b4e479b0e6 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:38.436891 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:38.483362 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a05f8f-3419-407e-85d4-013f87b9bbb2 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:38.484882 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a05f8f-3419-407e-85d4-013f87b9bbb2 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:38.485657 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a05f8f-3419-407e-85d4-013f87b9bbb2 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:38.495362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ad9031-1f9e-4ec4-9686-a1b4e479b0e6 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:38.497353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 168/341] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:38 2022] GET /identity => generated 270 bytes in 16 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:38.510637 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b577be78-608f-450a-95f3-78e79282aab3 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:38.511527 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b577be78-608f-450a-95f3-78e79282aab3 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:38.512294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b577be78-608f-450a-95f3-78e79282aab3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:38.532889 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:38.552086 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c7160b-ed17-47fd-9ca3-952e4a3a7990 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:38.552833 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c7160b-ed17-47fd-9ca3-952e4a3a7990 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:38.553115 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c7160b-ed17-47fd-9ca3-952e4a3a7990 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:38.553386 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c7160b-ed17-47fd-9ca3-952e4a3a7990 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:38.566251 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c7160b-ed17-47fd-9ca3-952e4a3a7990 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:38.567061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-b577be78-608f-450a-95f3-78e79282aab3 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:38.571327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-47c7160b-ed17-47fd-9ca3-952e4a3a7990 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:38.571327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:38.571327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:38.571327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:38.571327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:38.571327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:38.571327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:38.571327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:38.571327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:38.571327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:38.571327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:38.571327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:38.571327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:38.571327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:38.571327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:38.571327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:38.573113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:38.573113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:38.573113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:38.573113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:38.573113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:38.573113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:38.573113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:38.573113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:38.573113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:38.573113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:38.573113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:38.573113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:38.573113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:38.573113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:38.573113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:38.573113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:38.574699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:38.574699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:38.574699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:38.574699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:38.574699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:38.574699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:38.574699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:38.574699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:38.574699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:38.574699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:38.574699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:38.574699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:38.576819 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-b577be78-608f-450a-95f3-78e79282aab3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:38.577115 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 174/342] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:38.588009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c7160b-ed17-47fd-9ca3-952e4a3a7990 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:38.759266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c7160b-ed17-47fd-9ca3-952e4a3a7990 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:38.763256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 169/343] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:38.764281 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b577be78-608f-450a-95f3-78e79282aab3 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:38.827623 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:38.843232 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3e3bd4-a636-4219-b118-14908f2bdad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:38.844860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3e3bd4-a636-4219-b118-14908f2bdad1 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:38.845690 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3e3bd4-a636-4219-b118-14908f2bdad1 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:38.846558 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3e3bd4-a636-4219-b118-14908f2bdad1 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:38.849205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3e3bd4-a636-4219-b118-14908f2bdad1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:38.850093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3e3bd4-a636-4219-b118-14908f2bdad1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:38.852953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3e3bd4-a636-4219-b118-14908f2bdad1 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:38.870802 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 175/344] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:14:38.877398 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ce221f-6537-49a5-88fb-31e975b34c9c None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:38.879686 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ce221f-6537-49a5-88fb-31e975b34c9c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:38.880507 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ce221f-6537-49a5-88fb-31e975b34c9c None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:38.887917 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 176/345] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:38 2022] GET /identity => generated 270 bytes in 16 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:38.889764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252b0b81-9b67-4526-a62f-2f25d0886418 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:38.890389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252b0b81-9b67-4526-a62f-2f25d0886418 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:38.890813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252b0b81-9b67-4526-a62f-2f25d0886418 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:38.892854 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 177/346] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:38.900957 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252b0b81-9b67-4526-a62f-2f25d0886418 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:38.915948 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b577be78-608f-450a-95f3-78e79282aab3 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:38.976537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab4ef3a-ae32-45d5-ade6-69f9dcf234b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:38.977192 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab4ef3a-ae32-45d5-ade6-69f9dcf234b1 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:38.977670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab4ef3a-ae32-45d5-ade6-69f9dcf234b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:38.978097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab4ef3a-ae32-45d5-ade6-69f9dcf234b1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:38.980642 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:38.993097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab4ef3a-ae32-45d5-ade6-69f9dcf234b1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:38.998292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-9ab4ef3a-ae32-45d5-ade6-69f9dcf234b1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:38.998292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:38.998292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:38.998292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:38.998292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:38.998292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:38.998292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:38.998292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:38.998292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:38.998292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:38.998292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:38.998292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:38.998292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:38.998292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:38.998292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:38.998292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:38.999456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:38.999456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:38.999456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:38.999456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:38.999456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:38.999456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:38.999456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:38.999456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:38.999456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:38.999456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:38.999456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:38.999456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:38.999456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:38.999456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:38.999456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:38.999456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:39.000508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:39.000508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:39.000508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:39.000508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:39.000508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:39.000508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:39.000508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:39.000508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:39.000508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:39.000508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:39.000508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:39.000508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:39.002570 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 170/347] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:14:38 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:39.004164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fa6ad5-6ddf-467e-872b-e7349677aa91 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:39.004488 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fa6ad5-6ddf-467e-872b-e7349677aa91 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:39.004773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fa6ad5-6ddf-467e-872b-e7349677aa91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:39.025379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-f3fa6ad5-6ddf-467e-872b-e7349677aa91 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:39.026537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-f3fa6ad5-6ddf-467e-872b-e7349677aa91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:39.085426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252b0b81-9b67-4526-a62f-2f25d0886418 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:39.137745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fc5dd3-d433-4791-bb25-c65f11777f3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:39.138383 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fc5dd3-d433-4791-bb25-c65f11777f3b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:39.138661 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fc5dd3-d433-4791-bb25-c65f11777f3b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:39.138932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fc5dd3-d433-4791-bb25-c65f11777f3b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:39.166460 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 171/348] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:39.169973 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c56f69-520c-4971-9cb4-ef7f4bc88a58 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:39.170509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c56f69-520c-4971-9cb4-ef7f4bc88a58 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:39.170984 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c56f69-520c-4971-9cb4-ef7f4bc88a58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:39.206487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-e5c56f69-520c-4971-9cb4-ef7f4bc88a58 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:39.209702 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-e5c56f69-520c-4971-9cb4-ef7f4bc88a58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:39.245627 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:39.268056 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94fc5dd3-d433-4791-bb25-c65f11777f3b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:39.270570 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94fc5dd3-d433-4791-bb25-c65f11777f3b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:39.335083 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 172/349] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:39.336376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c56f69-520c-4971-9cb4-ef7f4bc88a58 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:39.470894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:39.484926 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c56f69-520c-4971-9cb4-ef7f4bc88a58 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:39.497698 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 178/350] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 604 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:14:39.499182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fc5dd3-d433-4791-bb25-c65f11777f3b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:39.565089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f200dd6-2ad4-4780-8f62-6df9265b444e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:39.566827 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f200dd6-2ad4-4780-8f62-6df9265b444e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:39.567312 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f200dd6-2ad4-4780-8f62-6df9265b444e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:39.567786 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f200dd6-2ad4-4780-8f62-6df9265b444e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:39.590665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f200dd6-2ad4-4780-8f62-6df9265b444e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:39.596780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-0f200dd6-2ad4-4780-8f62-6df9265b444e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:14:39.596780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:39.596780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:39.596780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:39.596780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:39.596780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:39.596780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:39.596780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:39.596780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:39.596780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:39.596780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:39.596780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:39.596780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:39.596780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:39.596780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:39.596780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:39.598710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:39.598710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:39.598710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:39.598710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:39.598710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:39.598710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:39.598710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:39.598710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:39.598710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:39.598710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:39.598710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:39.598710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:39.598710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:39.598710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:39.598710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:39.598710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:39.600285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:39.600285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:39.600285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:39.600285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:39.600285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:39.600285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:39.600285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:39.600285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:39.600285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:39.600285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:39.600285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:14:39.600285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:39.600285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 173/351] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:14:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:39.600285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f200dd6-2ad4-4780-8f62-6df9265b444e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:39.692434 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:39.703808 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fc5dd3-d433-4791-bb25-c65f11777f3b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:39.764115 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd35054f-58e6-4a40-9abe-4bf3885423c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:39.764788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd35054f-58e6-4a40-9abe-4bf3885423c8 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:39.766011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd35054f-58e6-4a40-9abe-4bf3885423c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:39.766421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd35054f-58e6-4a40-9abe-4bf3885423c8 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:39.767164 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd35054f-58e6-4a40-9abe-4bf3885423c8 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:39.767474 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd35054f-58e6-4a40-9abe-4bf3885423c8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:39.770084 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd35054f-58e6-4a40-9abe-4bf3885423c8 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:39.780133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 179/352] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:14:39 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:39.781125 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd35054f-58e6-4a40-9abe-4bf3885423c8 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:39.796539 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f200dd6-2ad4-4780-8f62-6df9265b444e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:39.821169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:39.862735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5cd5d28-3326-4072-99d6-5b113b5473b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:39.863719 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cd5d28-3326-4072-99d6-5b113b5473b4 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:39.865028 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cd5d28-3326-4072-99d6-5b113b5473b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:39.865028 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cd5d28-3326-4072-99d6-5b113b5473b4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:39.885685 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5cd5d28-3326-4072-99d6-5b113b5473b4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:39.892914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-a5cd5d28-3326-4072-99d6-5b113b5473b4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:39.892914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:39.892914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:39.892914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:39.892914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:39.892914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:39.892914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:39.892914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:39.892914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:39.892914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:39.892914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:39.892914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:39.892914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:39.892914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:39.892914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:39.892914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:39.895981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:39.895981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:39.895981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:39.895981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:39.895981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:39.895981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:39.895981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:39.895981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:39.895981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:39.895981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:39.895981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:39.895981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:39.895981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:39.895981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:39.895981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:39.895981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:39.898187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:39.898187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:39.898187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:39.898187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:39.898187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:39.898187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:39.898187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:39.898187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:39.898187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:39.898187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:39.898187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:39.898187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:39.898187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 174/353] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:39.898187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5cd5d28-3326-4072-99d6-5b113b5473b4 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:39.974959 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd35054f-58e6-4a40-9abe-4bf3885423c8 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:40.048253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f088f119-df05-4313-838a-c6954bc1d3f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:40.049403 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f088f119-df05-4313-838a-c6954bc1d3f4 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:40.051344 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f088f119-df05-4313-838a-c6954bc1d3f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:40.051801 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f088f119-df05-4313-838a-c6954bc1d3f4 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:40.072440 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f088f119-df05-4313-838a-c6954bc1d3f4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:40.080103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-f088f119-df05-4313-838a-c6954bc1d3f4 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:14:40.080103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:40.080103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:40.080103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:40.080103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:40.080103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:40.080103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:40.080103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:40.080103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:40.080103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:40.080103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:40.080103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:40.080103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:40.080103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:40.080103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:40.080103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:40.083517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:40.083517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:40.083517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:40.083517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:40.083517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:40.083517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:40.083517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:40.083517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:40.083517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:40.083517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:40.083517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:40.083517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:40.083517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:40.083517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:40.083517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:40.083517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:40.085075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:40.085075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:40.085075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:40.085075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:40.085075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:40.085075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:40.085075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:40.085075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:40.085075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:40.085075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:40.085075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 14:14:40.085075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:40.085075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 180/354] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:39 2022] GET /identity/v3/roles/reader => generated 84 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:40.085075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f088f119-df05-4313-838a-c6954bc1d3f4 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:40.091271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5cd5d28-3326-4072-99d6-5b113b5473b4 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:40.107635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:40.147300 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224e6b8c-cc3b-4539-94c8-edd41c912ef1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:40.147865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224e6b8c-cc3b-4539-94c8-edd41c912ef1 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:40.148154 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224e6b8c-cc3b-4539-94c8-edd41c912ef1 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:40.148455 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224e6b8c-cc3b-4539-94c8-edd41c912ef1 None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:40.149820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224e6b8c-cc3b-4539-94c8-edd41c912ef1 None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:40.151334 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224e6b8c-cc3b-4539-94c8-edd41c912ef1 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:40.153201 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 175/355] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:39 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:40.154051 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224e6b8c-cc3b-4539-94c8-edd41c912ef1 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:40.231593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:40.279910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f088f119-df05-4313-838a-c6954bc1d3f4 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:40.341303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224e6b8c-cc3b-4539-94c8-edd41c912ef1 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:40.342663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0769b90c-f182-4062-92c6-10e21801fbd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:40.343631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0769b90c-f182-4062-92c6-10e21801fbd2 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:40.344056 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0769b90c-f182-4062-92c6-10e21801fbd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:40.344612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0769b90c-f182-4062-92c6-10e21801fbd2 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:40.345630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0769b90c-f182-4062-92c6-10e21801fbd2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:40.346065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0769b90c-f182-4062-92c6-10e21801fbd2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:40.350304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0769b90c-f182-4062-92c6-10e21801fbd2 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:40.364706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 181/356] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:40.366050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0769b90c-f182-4062-92c6-10e21801fbd2 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:40.401179 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bc0f91-f267-4ad9-a774-0183aea7cfb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:40.401933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bc0f91-f267-4ad9-a774-0183aea7cfb4 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:40.402272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bc0f91-f267-4ad9-a774-0183aea7cfb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:40.402602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bc0f91-f267-4ad9-a774-0183aea7cfb4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:40.424122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0bc0f91-f267-4ad9-a774-0183aea7cfb4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:40.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-d0bc0f91-f267-4ad9-a774-0183aea7cfb4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:40.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:40.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:40.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:40.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:40.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:40.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:40.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:40.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:40.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:40.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:40.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:40.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:40.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:40.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:40.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:40.437056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:40.437056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:40.437056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:40.437056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:40.437056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:40.437056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:40.437056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:40.437056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:40.437056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:40.437056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:40.437056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:40.437056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:40.437056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:40.437056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:40.437056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:40.437056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:40.438516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:40.438516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:40.438516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:40.438516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:40.438516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:40.438516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:40.438516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:40.438516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:40.438516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:40.438516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:40.438516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:40.438516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:40.438516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 176/357] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:14:40 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:40.439285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bc0f91-f267-4ad9-a774-0183aea7cfb4 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:40.486438 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:40.541639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0769b90c-f182-4062-92c6-10e21801fbd2 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:40.608967 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec012e2-3768-42ee-8ca2-93297da41c7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:40.609610 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec012e2-3768-42ee-8ca2-93297da41c7a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:40.609887 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec012e2-3768-42ee-8ca2-93297da41c7a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:40.610163 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec012e2-3768-42ee-8ca2-93297da41c7a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:40.612727 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bc0f91-f267-4ad9-a774-0183aea7cfb4 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:40.614646 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:40.682342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683f3297-6599-4ac7-8960-cd5137a6b89f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:40.683215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683f3297-6599-4ac7-8960-cd5137a6b89f None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:40.683665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683f3297-6599-4ac7-8960-cd5137a6b89f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:40.684047 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683f3297-6599-4ac7-8960-cd5137a6b89f None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:40.685215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683f3297-6599-4ac7-8960-cd5137a6b89f None admin] RBAC: Adding query filter params (name=member) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:40.685594 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683f3297-6599-4ac7-8960-cd5137a6b89f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:40.687918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683f3297-6599-4ac7-8960-cd5137a6b89f None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:40.705725 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 177/358] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:40 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:40.708257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683f3297-6599-4ac7-8960-cd5137a6b89f None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:40.733406 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec012e2-3768-42ee-8ca2-93297da41c7a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:40.735778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec012e2-3768-42ee-8ca2-93297da41c7a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:40.851267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:40.868178 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683f3297-6599-4ac7-8960-cd5137a6b89f None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:40.903620 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 182/359] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 538 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:14:40.904956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec012e2-3768-42ee-8ca2-93297da41c7a None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:40.910972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b880b59-5777-49a5-aaa4-41483cf370ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:40.911885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b880b59-5777-49a5-aaa4-41483cf370ce None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:40.912285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b880b59-5777-49a5-aaa4-41483cf370ce None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:40.912722 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b880b59-5777-49a5-aaa4-41483cf370ce None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:40.914024 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b880b59-5777-49a5-aaa4-41483cf370ce None admin] RBAC: Adding query filter params (name=reader) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:40.914536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b880b59-5777-49a5-aaa4-41483cf370ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:40.916956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b880b59-5777-49a5-aaa4-41483cf370ce None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:40.933437 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 178/360] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:40 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:40.934381 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b880b59-5777-49a5-aaa4-41483cf370ce None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:40.978680 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:41.063965 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec012e2-3768-42ee-8ca2-93297da41c7a None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:41.092001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b880b59-5777-49a5-aaa4-41483cf370ce None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:41.110424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fa37ee-fb53-4f93-822a-a33424f1171b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:41.111350 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fa37ee-fb53-4f93-822a-a33424f1171b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:41.111757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fa37ee-fb53-4f93-822a-a33424f1171b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:41.112987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fa37ee-fb53-4f93-822a-a33424f1171b None admin] PATH_INFO: `/v3/groups/b657181658cf47efb236289d021c0f16` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:41.121084 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fa37ee-fb53-4f93-822a-a33424f1171b None admin] RBAC: Authorizing `identity:get_group(group_id=b657181658cf47efb236289d021c0f16)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:41.122942 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fa37ee-fb53-4f93-822a-a33424f1171b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:41.125176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 183/361] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Oct 4 14:14:40 2022] GET /identity/v3/groups/b657181658cf47efb236289d021c0f16 => generated 226 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:41.126550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fa37ee-fb53-4f93-822a-a33424f1171b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:41.146148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b790b02b-017b-4ebe-b744-dadfbac91d0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:41.146787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b790b02b-017b-4ebe-b744-dadfbac91d0e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:41.147070 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b790b02b-017b-4ebe-b744-dadfbac91d0e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:41.147345 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b790b02b-017b-4ebe-b744-dadfbac91d0e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:41.186992 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:41.228929 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b790b02b-017b-4ebe-b744-dadfbac91d0e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:41.230426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b790b02b-017b-4ebe-b744-dadfbac91d0e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:41.266266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:41.286822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fa37ee-fb53-4f93-822a-a33424f1171b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:41.336798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d719161-cbe1-430a-bd94-38007d2ed167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:41.337505 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d719161-cbe1-430a-bd94-38007d2ed167 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:41.337841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d719161-cbe1-430a-bd94-38007d2ed167 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:41.338186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d719161-cbe1-430a-bd94-38007d2ed167 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:41.339165 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d719161-cbe1-430a-bd94-38007d2ed167 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:41.339528 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d719161-cbe1-430a-bd94-38007d2ed167 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:41.341271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d719161-cbe1-430a-bd94-38007d2ed167 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:41.352075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 184/362] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:14:41 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:41.352751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d719161-cbe1-430a-bd94-38007d2ed167 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:41.372934 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 179/363] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:14:41.375003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b790b02b-017b-4ebe-b744-dadfbac91d0e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:41.501850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:41.508917 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d719161-cbe1-430a-bd94-38007d2ed167 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:41.532608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b790b02b-017b-4ebe-b744-dadfbac91d0e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:41.557040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01f81fa-a224-47f8-99b5-21bb1e2cef7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:41.557991 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01f81fa-a224-47f8-99b5-21bb1e2cef7f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:41.558332 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01f81fa-a224-47f8-99b5-21bb1e2cef7f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:41.559156 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01f81fa-a224-47f8-99b5-21bb1e2cef7f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:41.565698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:41.576346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb8b519-203f-4c0e-bed2-5f9be779f1ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:41.577014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb8b519-203f-4c0e-bed2-5f9be779f1ab None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:41.577298 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb8b519-203f-4c0e-bed2-5f9be779f1ab None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:41.577575 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb8b519-203f-4c0e-bed2-5f9be779f1ab None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:41.578868 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb8b519-203f-4c0e-bed2-5f9be779f1ab None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:41.580389 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb8b519-203f-4c0e-bed2-5f9be779f1ab None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:41.583330 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 180/364] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:41 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:41.583966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb8b519-203f-4c0e-bed2-5f9be779f1ab None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:41.648015 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e01f81fa-a224-47f8-99b5-21bb1e2cef7f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:41.650979 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e01f81fa-a224-47f8-99b5-21bb1e2cef7f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:41.746559 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb8b519-203f-4c0e-bed2-5f9be779f1ab None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:41.781665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:41.786881 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 185/365] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:14:41.787208 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01f81fa-a224-47f8-99b5-21bb1e2cef7f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:41.804268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fe7602-b166-4682-b4ea-f42cf3dede88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:41.805949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fe7602-b166-4682-b4ea-f42cf3dede88 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:41.806148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fe7602-b166-4682-b4ea-f42cf3dede88 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:41.806465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fe7602-b166-4682-b4ea-f42cf3dede88 None admin] PATH_INFO: `/v3/users/3783179711a2422fb709fa511638266e` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:41.809537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fe7602-b166-4682-b4ea-f42cf3dede88 None admin] RBAC: Authorizing `identity:get_user(user_id=3783179711a2422fb709fa511638266e)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:41.812279 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fe7602-b166-4682-b4ea-f42cf3dede88 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:41.821079 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 181/366] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:14:41 2022] GET /identity/v3/users/3783179711a2422fb709fa511638266e => generated 293 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:41.821079 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fe7602-b166-4682-b4ea-f42cf3dede88 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:41.903749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:41.943394 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01f81fa-a224-47f8-99b5-21bb1e2cef7f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:41.969312 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fe7602-b166-4682-b4ea-f42cf3dede88 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:42.008819 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3934554-0c14-42a9-8a5e-59609caa639c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:42.009712 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3934554-0c14-42a9-8a5e-59609caa639c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:42.010174 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3934554-0c14-42a9-8a5e-59609caa639c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:42.010624 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3934554-0c14-42a9-8a5e-59609caa639c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:42.011938 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3934554-0c14-42a9-8a5e-59609caa639c None admin] RBAC: Adding query filter params (group.id=b657181658cf47efb236289d021c0f16, role.id=bc6aeaa495ca4f3dbe72905c744c2233, scope.project.id=957c97404bb9448e9d03eee0d3225b85) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:42.012503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3934554-0c14-42a9-8a5e-59609caa639c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:42.015143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3934554-0c14-42a9-8a5e-59609caa639c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:42.022335 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ee59e0-414b-4529-b6b0-c9f58b851101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:42.022983 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ee59e0-414b-4529-b6b0-c9f58b851101 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:42.023335 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ee59e0-414b-4529-b6b0-c9f58b851101 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:42.023616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ee59e0-414b-4529-b6b0-c9f58b851101 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:42.038391 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 186/367] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Oct 4 14:14:41 2022] GET /identity/v3/role_assignments?group.id=b657181658cf47efb236289d021c0f16&scope.project.id=957c97404bb9448e9d03eee0d3225b85&role.id=bc6aeaa495ca4f3dbe72905c744c2233 => generated 260 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:42.038391 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3934554-0c14-42a9-8a5e-59609caa639c None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:42.134109 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ee59e0-414b-4529-b6b0-c9f58b851101 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:42.137017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ee59e0-414b-4529-b6b0-c9f58b851101 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:42.154047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:42.227050 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:42.238217 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3934554-0c14-42a9-8a5e-59609caa639c None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:42.294901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5407ae0c-d9bb-470b-8d2d-cee03de2c62c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:42.295445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 182/368] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 479 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:14:42.295601 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5407ae0c-d9bb-470b-8d2d-cee03de2c62c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:42.296041 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5407ae0c-d9bb-470b-8d2d-cee03de2c62c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:42.296569 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5407ae0c-d9bb-470b-8d2d-cee03de2c62c None admin] PATH_INFO: `/v3/projects/79763f634e4d4d53967547cda84e5ebd` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:42.297192 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ee59e0-414b-4529-b6b0-c9f58b851101 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:42.302368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5407ae0c-d9bb-470b-8d2d-cee03de2c62c None admin] RBAC: Authorizing `identity:get_project(project_id=79763f634e4d4d53967547cda84e5ebd)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:42.304869 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5407ae0c-d9bb-470b-8d2d-cee03de2c62c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:42.307100 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 187/369] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:42 2022] GET /identity/v3/projects/79763f634e4d4d53967547cda84e5ebd => generated 344 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:42.312531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5407ae0c-d9bb-470b-8d2d-cee03de2c62c None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:42.464793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ee59e0-414b-4529-b6b0-c9f58b851101 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:42.519211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5407ae0c-d9bb-470b-8d2d-cee03de2c62c None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:42.528299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12dca60-48cd-4cda-96f5-c67b4418bba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:42.529343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12dca60-48cd-4cda-96f5-c67b4418bba6 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:42.529800 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12dca60-48cd-4cda-96f5-c67b4418bba6 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:42.530250 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12dca60-48cd-4cda-96f5-c67b4418bba6 None admin] PATH_INFO: `/v3/groups/b657181658cf47efb236289d021c0f16` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:42.532509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12dca60-48cd-4cda-96f5-c67b4418bba6 None admin] RBAC: Authorizing `identity:get_group(group_id=b657181658cf47efb236289d021c0f16)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:42.535360 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12dca60-48cd-4cda-96f5-c67b4418bba6 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:42.538663 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 183/370] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Oct 4 14:14:42 2022] GET /identity/v3/groups/b657181658cf47efb236289d021c0f16 => generated 226 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:42.540114 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12dca60-48cd-4cda-96f5-c67b4418bba6 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:42.584540 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:42.595478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe8c055-e9e1-416a-ab45-3284e6992fb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:42.596645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe8c055-e9e1-416a-ab45-3284e6992fb6 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:42.597141 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe8c055-e9e1-416a-ab45-3284e6992fb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:42.597594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe8c055-e9e1-416a-ab45-3284e6992fb6 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:42.598363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:42.599808 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe8c055-e9e1-416a-ab45-3284e6992fb6 None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:42.602267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe8c055-e9e1-416a-ab45-3284e6992fb6 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:42.605057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 188/371] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:42 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:42.606400 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe8c055-e9e1-416a-ab45-3284e6992fb6 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:42.726331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12dca60-48cd-4cda-96f5-c67b4418bba6 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:42.777665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7132d6da-8350-457e-84a1-18107b6fd751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:42.778681 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7132d6da-8350-457e-84a1-18107b6fd751 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:42.778681 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7132d6da-8350-457e-84a1-18107b6fd751 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:42.778943 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7132d6da-8350-457e-84a1-18107b6fd751 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:42.779583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7132d6da-8350-457e-84a1-18107b6fd751 None admin] RBAC: Adding query filter params (role.id=f425b98e3acf4e13b0da24b21e59cfce, user.id=3783179711a2422fb709fa511638266e, scope.system=all) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:42.779884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7132d6da-8350-457e-84a1-18107b6fd751 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:42.781752 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7132d6da-8350-457e-84a1-18107b6fd751 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:42.804379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 184/372] 10.222.0.64 () {56 vars in 1308 bytes} [Tue Oct 4 14:14:42 2022] GET /identity/v3/role_assignments?user.id=3783179711a2422fb709fa511638266e&scope.system=all&role.id=f425b98e3acf4e13b0da24b21e59cfce => generated 510 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:42.804379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7132d6da-8350-457e-84a1-18107b6fd751 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:42.811760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe8c055-e9e1-416a-ab45-3284e6992fb6 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:42.862810 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5297fe25-0556-46d9-97a8-f00e65f2eac8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:42.863820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5297fe25-0556-46d9-97a8-f00e65f2eac8 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:42.864104 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5297fe25-0556-46d9-97a8-f00e65f2eac8 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:42.864415 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5297fe25-0556-46d9-97a8-f00e65f2eac8 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:42.866460 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5297fe25-0556-46d9-97a8-f00e65f2eac8 None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:42.868374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5297fe25-0556-46d9-97a8-f00e65f2eac8 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:42.870984 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 189/373] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:42 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:42.872716 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5297fe25-0556-46d9-97a8-f00e65f2eac8 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:42.975888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:42.995942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:43.027534 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7132d6da-8350-457e-84a1-18107b6fd751 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:43.035864 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5297fe25-0556-46d9-97a8-f00e65f2eac8 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:43.087603 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e4b622-6509-4588-aa8a-ec5193ca9404 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:43.089196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4b622-6509-4588-aa8a-ec5193ca9404 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:43.089664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4b622-6509-4588-aa8a-ec5193ca9404 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:43.090387 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4b622-6509-4588-aa8a-ec5193ca9404 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:43.091364 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e4b622-6509-4588-aa8a-ec5193ca9404 None admin] RBAC: Adding query filter params (group.id=b657181658cf47efb236289d021c0f16, role.id=e114d28e2dd147fcb39452dd461791c0, scope.project.id=957c97404bb9448e9d03eee0d3225b85) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:43.091673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e4b622-6509-4588-aa8a-ec5193ca9404 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:43.093229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e4b622-6509-4588-aa8a-ec5193ca9404 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:43.100614 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082511bd-f92b-4305-93d8-ea2d8ba51139 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:43.101768 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 190/374] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Oct 4 14:14:42 2022] GET /identity/v3/role_assignments?group.id=b657181658cf47efb236289d021c0f16&scope.project.id=957c97404bb9448e9d03eee0d3225b85&role.id=e114d28e2dd147fcb39452dd461791c0 => generated 260 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:43.102528 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082511bd-f92b-4305-93d8-ea2d8ba51139 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:43.102686 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e4b622-6509-4588-aa8a-ec5193ca9404 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:43.103477 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082511bd-f92b-4305-93d8-ea2d8ba51139 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:43.104213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082511bd-f92b-4305-93d8-ea2d8ba51139 None admin] PATH_INFO: `/v3/groups/63b6321f919240fd87c9ccafb29b636c` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:43.117860 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082511bd-f92b-4305-93d8-ea2d8ba51139 None admin] RBAC: Authorizing `identity:get_group(group_id=63b6321f919240fd87c9ccafb29b636c)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:43.121440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082511bd-f92b-4305-93d8-ea2d8ba51139 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:43.125888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 185/375] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Oct 4 14:14:42 2022] GET /identity/v3/groups/63b6321f919240fd87c9ccafb29b636c => generated 229 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:43.128843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082511bd-f92b-4305-93d8-ea2d8ba51139 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:43.284673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082511bd-f92b-4305-93d8-ea2d8ba51139 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:43.288689 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e4b622-6509-4588-aa8a-ec5193ca9404 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:43.328125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77025d9d-018d-42a5-a2bd-44a80d6fb7ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:43.329215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77025d9d-018d-42a5-a2bd-44a80d6fb7ab None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:43.329656 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77025d9d-018d-42a5-a2bd-44a80d6fb7ab None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:43.330079 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77025d9d-018d-42a5-a2bd-44a80d6fb7ab None admin] PATH_INFO: `/v3/groups/b657181658cf47efb236289d021c0f16` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:43.332366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77025d9d-018d-42a5-a2bd-44a80d6fb7ab None admin] RBAC: Authorizing `identity:get_group(group_id=b657181658cf47efb236289d021c0f16)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:43.334174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77025d9d-018d-42a5-a2bd-44a80d6fb7ab None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:43.336232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 186/376] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Oct 4 14:14:43 2022] GET /identity/v3/groups/b657181658cf47efb236289d021c0f16 => generated 226 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:43.337303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77025d9d-018d-42a5-a2bd-44a80d6fb7ab None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:43.355480 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:43.370454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:43.371483 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adce10cb-ee09-4710-ae61-e4fc16a7a629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:43.374597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adce10cb-ee09-4710-ae61-e4fc16a7a629 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:43.375501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adce10cb-ee09-4710-ae61-e4fc16a7a629 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:43.376287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adce10cb-ee09-4710-ae61-e4fc16a7a629 None admin] PATH_INFO: `/v3/groups/b657181658cf47efb236289d021c0f16` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:43.385087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adce10cb-ee09-4710-ae61-e4fc16a7a629 None admin] RBAC: Authorizing `identity:get_group(group_id=b657181658cf47efb236289d021c0f16)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:43.388948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adce10cb-ee09-4710-ae61-e4fc16a7a629 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:43.392786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 191/377] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Oct 4 14:14:43 2022] GET /identity/v3/groups/b657181658cf47efb236289d021c0f16 => generated 226 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:43.400057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adce10cb-ee09-4710-ae61-e4fc16a7a629 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:43.485134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77025d9d-018d-42a5-a2bd-44a80d6fb7ab None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:43.570483 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855ae674-560e-4bce-81cd-83b76c70f625 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:43.571037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855ae674-560e-4bce-81cd-83b76c70f625 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:43.571626 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855ae674-560e-4bce-81cd-83b76c70f625 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:43.572202 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855ae674-560e-4bce-81cd-83b76c70f625 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:43.573672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855ae674-560e-4bce-81cd-83b76c70f625 None admin] RBAC: Adding query filter params (group.id=63b6321f919240fd87c9ccafb29b636c, role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=79763f634e4d4d53967547cda84e5ebd) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:43.574317 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855ae674-560e-4bce-81cd-83b76c70f625 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:43.577377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855ae674-560e-4bce-81cd-83b76c70f625 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:43.589422 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 187/378] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Oct 4 14:14:43 2022] GET /identity/v3/role_assignments?group.id=63b6321f919240fd87c9ccafb29b636c&scope.project.id=79763f634e4d4d53967547cda84e5ebd&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 260 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:43.590696 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855ae674-560e-4bce-81cd-83b76c70f625 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:43.649271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adce10cb-ee09-4710-ae61-e4fc16a7a629 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:43.763817 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c6523d-3da9-433b-b56e-9043d9188529 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:43.765228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c6523d-3da9-433b-b56e-9043d9188529 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:43.765954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c6523d-3da9-433b-b56e-9043d9188529 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:43.766625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c6523d-3da9-433b-b56e-9043d9188529 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:43.768813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c6523d-3da9-433b-b56e-9043d9188529 None admin] RBAC: Adding query filter params (group.id=b657181658cf47efb236289d021c0f16, role.id=e114d28e2dd147fcb39452dd461791c0, scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:43.769718 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c6523d-3da9-433b-b56e-9043d9188529 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:43.772685 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c6523d-3da9-433b-b56e-9043d9188529 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:43.793859 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 192/379] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Oct 4 14:14:43 2022] GET /identity/v3/role_assignments?group.id=b657181658cf47efb236289d021c0f16&scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535&role.id=e114d28e2dd147fcb39452dd461791c0 => generated 260 bytes in 400 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:43.848841 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:43.874698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855ae674-560e-4bce-81cd-83b76c70f625 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:43.923413 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336a168a-07b5-4ca9-bb83-a64de301aac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:43.924067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336a168a-07b5-4ca9-bb83-a64de301aac5 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:43.924626 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336a168a-07b5-4ca9-bb83-a64de301aac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:43.925029 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336a168a-07b5-4ca9-bb83-a64de301aac5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:43.926095 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336a168a-07b5-4ca9-bb83-a64de301aac5 None admin] RBAC: Adding query filter params (group.id=b657181658cf47efb236289d021c0f16, role.id=bc6aeaa495ca4f3dbe72905c744c2233, scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:43.926414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336a168a-07b5-4ca9-bb83-a64de301aac5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:43.927991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336a168a-07b5-4ca9-bb83-a64de301aac5 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:43.940834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 188/380] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Oct 4 14:14:43 2022] GET /identity/v3/role_assignments?group.id=b657181658cf47efb236289d021c0f16&scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535&role.id=bc6aeaa495ca4f3dbe72905c744c2233 => generated 260 bytes in 351 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:45.445051 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177bc172-d124-4224-acad-f143c6e7b0a4 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:45.445839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177bc172-d124-4224-acad-f143c6e7b0a4 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:45.445839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177bc172-d124-4224-acad-f143c6e7b0a4 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:45.447277 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 193/381] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:45.454440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701a0b4a-c163-434d-be3c-6d0a075c9b25 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:45.457144 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701a0b4a-c163-434d-be3c-6d0a075c9b25 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:45.457144 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701a0b4a-c163-434d-be3c-6d0a075c9b25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:45.496433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-701a0b4a-c163-434d-be3c-6d0a075c9b25 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:45.498423 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-701a0b4a-c163-434d-be3c-6d0a075c9b25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:45.642460 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 189/382] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:46.194807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177bc172-d124-4224-acad-f143c6e7b0a4 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:46.300936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:46.367036 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177bc172-d124-4224-acad-f143c6e7b0a4 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:46.427633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0682a9-f7a3-442e-b77d-3e886a545e86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:46.428295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0682a9-f7a3-442e-b77d-3e886a545e86 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:46.428733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0682a9-f7a3-442e-b77d-3e886a545e86 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:46.429001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0682a9-f7a3-442e-b77d-3e886a545e86 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:46.456486 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0682a9-f7a3-442e-b77d-3e886a545e86 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:46.460906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-6d0682a9-f7a3-442e-b77d-3e886a545e86 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:46.460906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:46.460906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:46.460906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:46.460906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:46.460906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:46.460906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:46.460906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:46.460906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:46.460906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:46.460906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:46.460906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:46.460906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:46.460906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:46.460906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:46.460906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:46.462586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:46.462586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:46.462586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:46.462586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:46.462586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:46.462586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:46.462586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:46.462586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:46.462586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:46.462586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:46.462586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:46.462586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:46.462586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:46.462586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:46.462586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:46.462586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:46.465901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:46.465901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:46.465901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:46.465901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:46.465901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:46.465901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:46.465901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:46.465901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:46.465901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:46.465901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:46.465901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:46.465901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:46.465901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 194/383] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:46 2022] GET /identity/v3/roles/member => generated 84 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:46.472800 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701a0b4a-c163-434d-be3c-6d0a075c9b25 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:46.649143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:46.695375 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701a0b4a-c163-434d-be3c-6d0a075c9b25 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:46.771142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4c3b37-83a8-4cf0-bad0-6fe3ceec21e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:46.772110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4c3b37-83a8-4cf0-bad0-6fe3ceec21e7 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:46.772646 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4c3b37-83a8-4cf0-bad0-6fe3ceec21e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:46.773137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4c3b37-83a8-4cf0-bad0-6fe3ceec21e7 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:46.774401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4c3b37-83a8-4cf0-bad0-6fe3ceec21e7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:46.775174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4c3b37-83a8-4cf0-bad0-6fe3ceec21e7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:46.777553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4c3b37-83a8-4cf0-bad0-6fe3ceec21e7 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:46.802192 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 190/384] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:46 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 334 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:46.807745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0682a9-f7a3-442e-b77d-3e886a545e86 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:46.845667 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea530552-8c19-4563-b4f4-82d398833c99 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:46.846679 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea530552-8c19-4563-b4f4-82d398833c99 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:46.847140 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea530552-8c19-4563-b4f4-82d398833c99 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:46.851845 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 191/385] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:46 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:14:46.860889 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f98f73e-eb63-434a-af28-2ae845c33260 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:46.861424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f98f73e-eb63-434a-af28-2ae845c33260 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:46.862380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f98f73e-eb63-434a-af28-2ae845c33260 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:46.921689 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-6f98f73e-eb63-434a-af28-2ae845c33260 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:46.923589 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-6f98f73e-eb63-434a-af28-2ae845c33260 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:47.007305 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0682a9-f7a3-442e-b77d-3e886a545e86 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:47.082382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:47.129835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec01501-9776-4aa3-91e9-87a586b8bf36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:47.131450 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec01501-9776-4aa3-91e9-87a586b8bf36 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:47.132558 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec01501-9776-4aa3-91e9-87a586b8bf36 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:47.134917 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec01501-9776-4aa3-91e9-87a586b8bf36 None admin] PATH_INFO: `/v3/groups/b657181658cf47efb236289d021c0f16` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:47.138971 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec01501-9776-4aa3-91e9-87a586b8bf36 None admin] RBAC: Authorizing `identity:get_group(group_id=b657181658cf47efb236289d021c0f16)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:47.142760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec01501-9776-4aa3-91e9-87a586b8bf36 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:47.151486 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 195/386] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Oct 4 14:14:46 2022] GET /identity/v3/groups/b657181658cf47efb236289d021c0f16 => generated 226 bytes in 344 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:47.157763 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec01501-9776-4aa3-91e9-87a586b8bf36 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:47.171450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 192/387] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 313 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:14:47.286501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7216532-7052-4a09-83dd-cb79b4de429f None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:47.286784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7216532-7052-4a09-83dd-cb79b4de429f None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:47.287109 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7216532-7052-4a09-83dd-cb79b4de429f None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:47.289345 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 193/388] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:14:47.295438 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738bffc4-59c4-4326-b770-70598b096654 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:47.295733 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738bffc4-59c4-4326-b770-70598b096654 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:47.296060 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738bffc4-59c4-4326-b770-70598b096654 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:47.318211 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-738bffc4-59c4-4326-b770-70598b096654 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:47.320012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-738bffc4-59c4-4326-b770-70598b096654 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:47.422285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:47.423936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec01501-9776-4aa3-91e9-87a586b8bf36 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:47.434471 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 194/389] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:47.437094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b22a5ba-d77a-4a68-afb1-26b68586f505 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:47.437410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b22a5ba-d77a-4a68-afb1-26b68586f505 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:47.437644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b22a5ba-d77a-4a68-afb1-26b68586f505 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:47.445072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 195/390] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:47.447807 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9876d1-585e-4dc9-9cbc-35c67f855c00 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:47.448130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9876d1-585e-4dc9-9cbc-35c67f855c00 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:47.448401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9876d1-585e-4dc9-9cbc-35c67f855c00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:47.479875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400e852c-24bd-4d30-9373-c2ffb48a6b68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:47.481317 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400e852c-24bd-4d30-9373-c2ffb48a6b68 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:47.483086 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400e852c-24bd-4d30-9373-c2ffb48a6b68 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:47.483779 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400e852c-24bd-4d30-9373-c2ffb48a6b68 None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:47.489886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-bc9876d1-585e-4dc9-9cbc-35c67f855c00 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:47.491323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400e852c-24bd-4d30-9373-c2ffb48a6b68 None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:47.493069 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400e852c-24bd-4d30-9373-c2ffb48a6b68 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:47.496443 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 196/391] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:47 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 339 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:47.496884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-bc9876d1-585e-4dc9-9cbc-35c67f855c00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:47.508036 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400e852c-24bd-4d30-9373-c2ffb48a6b68 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:47.595178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:47.650563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400e852c-24bd-4d30-9373-c2ffb48a6b68 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:47.661123 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 196/392] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:47.694788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317e6c73-bb8f-4726-8893-62617e81a863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:47.695851 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317e6c73-bb8f-4726-8893-62617e81a863 None admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:47.696375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317e6c73-bb8f-4726-8893-62617e81a863 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:47.696934 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317e6c73-bb8f-4726-8893-62617e81a863 None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85/groups/b657181658cf47efb236289d021c0f16/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:47.699633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317e6c73-bb8f-4726-8893-62617e81a863 None admin] RBAC: Authorizing `identity:create_grant(project_id=957c97404bb9448e9d03eee0d3225b85, group_id=b657181658cf47efb236289d021c0f16, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:47.700444 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9876d1-585e-4dc9-9cbc-35c67f855c00 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:47.701680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317e6c73-bb8f-4726-8893-62617e81a863 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:47.727888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 197/393] 10.222.0.64 () {58 vars in 1442 bytes} [Tue Oct 4 14:14:47 2022] PUT /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85/groups/b657181658cf47efb236289d021c0f16/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:14:47.887463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317e6c73-bb8f-4726-8893-62617e81a863 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:47.946877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9876d1-585e-4dc9-9cbc-35c67f855c00 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:48.014951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:48.026215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7c3b32-d6b6-4b10-9e0c-e6af5d54bdf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:48.027273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7c3b32-d6b6-4b10-9e0c-e6af5d54bdf8 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:48.027757 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7c3b32-d6b6-4b10-9e0c-e6af5d54bdf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:48.028225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7c3b32-d6b6-4b10-9e0c-e6af5d54bdf8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:48.050091 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7c3b32-d6b6-4b10-9e0c-e6af5d54bdf8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:48.059932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-ba7c3b32-d6b6-4b10-9e0c-e6af5d54bdf8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:48.059932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:48.059932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:48.059932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:48.059932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:48.059932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:48.059932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:48.059932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:48.059932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:48.059932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:48.059932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:48.059932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:48.059932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:48.059932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:48.059932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:48.059932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:48.061138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:48.061138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:48.061138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:48.061138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:48.061138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:48.061138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:48.061138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:48.061138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:48.061138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:48.061138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:48.061138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:48.061138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:48.061138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:48.061138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:48.061138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:48.061138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:48.062648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:48.062648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:48.062648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:48.062648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:48.062648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:48.062648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:48.062648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:48.062648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:48.062648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:48.062648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:48.062648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:48.062648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:48.062648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 197/394] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:14:47 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 363 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:48.065669 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebfab4b-2fa1-43d2-b09f-eb774fab1ae2 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:48.065669 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebfab4b-2fa1-43d2-b09f-eb774fab1ae2 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:48.066391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebfab4b-2fa1-43d2-b09f-eb774fab1ae2 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:48.068150 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317e6c73-bb8f-4726-8893-62617e81a863 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:48.070620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 198/395] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:48 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:48.071618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebfab4b-2fa1-43d2-b09f-eb774fab1ae2 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:48.106086 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:48.130789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628c7884-a541-4dda-ace3-69522cf28e42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:48.131526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628c7884-a541-4dda-ace3-69522cf28e42 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:48.131781 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628c7884-a541-4dda-ace3-69522cf28e42 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:48.132071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628c7884-a541-4dda-ace3-69522cf28e42 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:48.147618 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628c7884-a541-4dda-ace3-69522cf28e42 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:48.153389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-628c7884-a541-4dda-ace3-69522cf28e42 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:48.153389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:48.153389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:48.153389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:48.153389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:48.153389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:48.153389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:48.153389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:48.153389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:48.153389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:48.153389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:48.153389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:48.153389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:48.153389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:48.153389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:48.153389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:48.154910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:48.154910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:48.154910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:48.154910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:48.154910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:48.154910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:48.154910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:48.154910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:48.154910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:48.154910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:48.154910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:48.154910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:48.154910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:48.154910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:48.154910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:48.154910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:48.156784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:48.156784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:48.156784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:48.156784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:48.156784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:48.156784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:48.156784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:48.156784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:48.156784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:48.156784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:48.156784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:48.156784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:48.156784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 198/396] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:48.156784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628c7884-a541-4dda-ace3-69522cf28e42 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:48.224197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebfab4b-2fa1-43d2-b09f-eb774fab1ae2 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:48.272428 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed123ccc-44a3-4665-bf93-062ea0fbb477 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:48.273077 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed123ccc-44a3-4665-bf93-062ea0fbb477 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:48.273353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed123ccc-44a3-4665-bf93-062ea0fbb477 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:48.273639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed123ccc-44a3-4665-bf93-062ea0fbb477 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:48.287944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed123ccc-44a3-4665-bf93-062ea0fbb477 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:48.293226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-ed123ccc-44a3-4665-bf93-062ea0fbb477 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:14:48.293226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:48.293226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:48.293226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:48.293226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:48.293226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:48.293226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:48.293226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:48.293226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:48.293226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:48.293226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:48.293226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:48.293226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:48.293226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:48.293226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:48.293226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:48.295732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:48.295732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:48.295732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:48.295732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:48.295732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:48.295732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:48.295732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:48.295732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:48.295732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:48.295732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:48.295732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:48.295732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:48.295732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:48.295732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:48.295732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:48.295732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:48.297416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:48.297416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:48.297416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:48.297416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:48.297416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:48.297416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:48.297416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:48.297416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:48.297416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:48.297416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:48.297416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:14:48.297416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:48.297416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 199/397] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:14:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:48.298529 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628c7884-a541-4dda-ace3-69522cf28e42 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:48.301398 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71eae84b-8bf5-4a15-b355-5d5c0c724727 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:48.301739 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71eae84b-8bf5-4a15-b355-5d5c0c724727 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:48.301975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71eae84b-8bf5-4a15-b355-5d5c0c724727 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:48.328091 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-71eae84b-8bf5-4a15-b355-5d5c0c724727 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:48.330128 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-71eae84b-8bf5-4a15-b355-5d5c0c724727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:48.346657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9b82ac-0d2b-4b01-a91d-67b1b4c2c403 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:48.347546 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9b82ac-0d2b-4b01-a91d-67b1b4c2c403 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:48.347833 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9b82ac-0d2b-4b01-a91d-67b1b4c2c403 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:48.348091 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9b82ac-0d2b-4b01-a91d-67b1b4c2c403 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:48.348860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9b82ac-0d2b-4b01-a91d-67b1b4c2c403 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:48.349165 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9b82ac-0d2b-4b01-a91d-67b1b4c2c403 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:48.350642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9b82ac-0d2b-4b01-a91d-67b1b4c2c403 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:48.363170 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 199/398] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:14:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:48.363915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9b82ac-0d2b-4b01-a91d-67b1b4c2c403 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:48.422681 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:48.433862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:48.480447 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 200/399] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:48.480447 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71eae84b-8bf5-4a15-b355-5d5c0c724727 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:48.535669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9b82ac-0d2b-4b01-a91d-67b1b4c2c403 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:48.608930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b50a3e1-a3e6-4253-93be-f5f46c56f10c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:48.610278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b50a3e1-a3e6-4253-93be-f5f46c56f10c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:48.610935 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b50a3e1-a3e6-4253-93be-f5f46c56f10c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:48.611502 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b50a3e1-a3e6-4253-93be-f5f46c56f10c None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:48.613674 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b50a3e1-a3e6-4253-93be-f5f46c56f10c None admin] RBAC: Adding query filter params (name=member) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:48.614295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b50a3e1-a3e6-4253-93be-f5f46c56f10c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:48.617555 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b50a3e1-a3e6-4253-93be-f5f46c56f10c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:48.644753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 200/400] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:48 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:48.646819 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b50a3e1-a3e6-4253-93be-f5f46c56f10c None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:48.670443 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71eae84b-8bf5-4a15-b355-5d5c0c724727 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:48.733879 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649fb7a3-26e6-496a-bea4-b9a86874c5bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:48.734419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649fb7a3-26e6-496a-bea4-b9a86874c5bd None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:48.734703 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649fb7a3-26e6-496a-bea4-b9a86874c5bd None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:48.734978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649fb7a3-26e6-496a-bea4-b9a86874c5bd None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:48.735698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649fb7a3-26e6-496a-bea4-b9a86874c5bd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:48.736009 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649fb7a3-26e6-496a-bea4-b9a86874c5bd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:48.738561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649fb7a3-26e6-496a-bea4-b9a86874c5bd None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:48.751418 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 201/401] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:48.752039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649fb7a3-26e6-496a-bea4-b9a86874c5bd None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:48.817479 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:48.877464 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b50a3e1-a3e6-4253-93be-f5f46c56f10c None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:48.879563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:48.939918 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05689968-de82-468e-94f9-6215b6b51e3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:48.940962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05689968-de82-468e-94f9-6215b6b51e3a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:48.941250 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05689968-de82-468e-94f9-6215b6b51e3a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:48.941519 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05689968-de82-468e-94f9-6215b6b51e3a None admin] PATH_INFO: `/v3/groups/b657181658cf47efb236289d021c0f16` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:48.943060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05689968-de82-468e-94f9-6215b6b51e3a None admin] RBAC: Authorizing `identity:get_group(group_id=b657181658cf47efb236289d021c0f16)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:48.944120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649fb7a3-26e6-496a-bea4-b9a86874c5bd None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:48.944666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05689968-de82-468e-94f9-6215b6b51e3a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:48.946522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 201/402] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Oct 4 14:14:48 2022] GET /identity/v3/groups/b657181658cf47efb236289d021c0f16 => generated 226 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:48.947456 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05689968-de82-468e-94f9-6215b6b51e3a None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:49.004353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cfca44-da34-424a-842e-fc14b531887e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:49.004938 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cfca44-da34-424a-842e-fc14b531887e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:49.005222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cfca44-da34-424a-842e-fc14b531887e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:49.005502 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cfca44-da34-424a-842e-fc14b531887e None admin] PATH_INFO: `/v3/groups/b657181658cf47efb236289d021c0f16` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:49.007393 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cfca44-da34-424a-842e-fc14b531887e None admin] RBAC: Authorizing `identity:get_group(group_id=b657181658cf47efb236289d021c0f16)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:49.008591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cfca44-da34-424a-842e-fc14b531887e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:49.015369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 202/403] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Oct 4 14:14:48 2022] GET /identity/v3/groups/b657181658cf47efb236289d021c0f16 => generated 226 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:49.015369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cfca44-da34-424a-842e-fc14b531887e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:49.137696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05689968-de82-468e-94f9-6215b6b51e3a None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:49.200529 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c9b81c-a50c-43cf-bea6-b9ef97d0ac18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:49.201544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c9b81c-a50c-43cf-bea6-b9ef97d0ac18 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:49.202078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c9b81c-a50c-43cf-bea6-b9ef97d0ac18 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:49.202518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c9b81c-a50c-43cf-bea6-b9ef97d0ac18 None admin] PATH_INFO: `/v3/groups/63b6321f919240fd87c9ccafb29b636c` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:49.206249 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c9b81c-a50c-43cf-bea6-b9ef97d0ac18 None admin] RBAC: Authorizing `identity:get_group(group_id=63b6321f919240fd87c9ccafb29b636c)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:49.210841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c9b81c-a50c-43cf-bea6-b9ef97d0ac18 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:49.211550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cfca44-da34-424a-842e-fc14b531887e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:49.221994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 202/404] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Oct 4 14:14:48 2022] GET /identity/v3/groups/63b6321f919240fd87c9ccafb29b636c => generated 229 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:49.221994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c9b81c-a50c-43cf-bea6-b9ef97d0ac18 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:49.238324 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:49.282405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d24c66e-e51d-4b1d-ac33-10a78108f908 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:49.283881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d24c66e-e51d-4b1d-ac33-10a78108f908 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:49.284284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d24c66e-e51d-4b1d-ac33-10a78108f908 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:49.284754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d24c66e-e51d-4b1d-ac33-10a78108f908 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:49.301380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:49.305995 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d24c66e-e51d-4b1d-ac33-10a78108f908 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:49.317076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-1d24c66e-e51d-4b1d-ac33-10a78108f908 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:49.317076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:49.317076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:49.317076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:49.317076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:49.317076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:49.317076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:49.317076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:49.317076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:49.317076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:49.317076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:49.317076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:49.317076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:49.317076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:49.317076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:49.317076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:49.318148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:49.318148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:49.318148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:49.318148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:49.318148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:49.318148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:49.318148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:49.318148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:49.318148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:49.318148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:49.318148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:49.318148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:49.318148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:49.318148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:49.318148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:49.318148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:49.319101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:49.319101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:49.319101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:49.319101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:49.319101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:49.319101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:49.319101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:49.319101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:49.319101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:49.319101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:49.319101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:49.319101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:49.319101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 203/405] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:14:49 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 308 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:49.320567 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d24c66e-e51d-4b1d-ac33-10a78108f908 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:49.414057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c9b81c-a50c-43cf-bea6-b9ef97d0ac18 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:49.482372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0328e9f4-234a-41d9-836d-0ada2ca05455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:49.483470 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0328e9f4-234a-41d9-836d-0ada2ca05455 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:49.483945 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0328e9f4-234a-41d9-836d-0ada2ca05455 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:49.484481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0328e9f4-234a-41d9-836d-0ada2ca05455 None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:49.487329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0328e9f4-234a-41d9-836d-0ada2ca05455 None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:49.490179 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0328e9f4-234a-41d9-836d-0ada2ca05455 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:49.498179 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 203/406] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:49 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:49.498179 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0328e9f4-234a-41d9-836d-0ada2ca05455 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:49.545496 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d24c66e-e51d-4b1d-ac33-10a78108f908 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:49.611749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dded432-0319-4dbf-bc77-072986682298 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:49.613039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dded432-0319-4dbf-bc77-072986682298 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:49.613537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dded432-0319-4dbf-bc77-072986682298 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:49.614028 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dded432-0319-4dbf-bc77-072986682298 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:49.616175 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dded432-0319-4dbf-bc77-072986682298 None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:49.619019 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dded432-0319-4dbf-bc77-072986682298 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:49.622578 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 204/407] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:49 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:49.624012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dded432-0319-4dbf-bc77-072986682298 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:49.681633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0328e9f4-234a-41d9-836d-0ada2ca05455 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:49.697235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:49.707055 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:49.739810 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a034a73b-29ca-44c4-89d8-a041a337a79c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:49.740971 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a034a73b-29ca-44c4-89d8-a041a337a79c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:49.741398 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a034a73b-29ca-44c4-89d8-a041a337a79c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:49.742303 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a034a73b-29ca-44c4-89d8-a041a337a79c None admin] PATH_INFO: `/v3/projects/79763f634e4d4d53967547cda84e5ebd` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:49.744043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a034a73b-29ca-44c4-89d8-a041a337a79c None admin] RBAC: Authorizing `identity:get_project(project_id=79763f634e4d4d53967547cda84e5ebd)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:49.745952 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a034a73b-29ca-44c4-89d8-a041a337a79c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:49.749122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 204/408] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:49 2022] GET /identity/v3/projects/79763f634e4d4d53967547cda84e5ebd => generated 344 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:49.750202 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a034a73b-29ca-44c4-89d8-a041a337a79c None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:49.827575 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dded432-0319-4dbf-bc77-072986682298 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:49.897843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19da74a-c871-40ba-8322-18c95a7895e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:49.899495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19da74a-c871-40ba-8322-18c95a7895e8 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:49.904333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19da74a-c871-40ba-8322-18c95a7895e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:49.905327 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19da74a-c871-40ba-8322-18c95a7895e8 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:49.907086 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19da74a-c871-40ba-8322-18c95a7895e8 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:49.910707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19da74a-c871-40ba-8322-18c95a7895e8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:49.912176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19da74a-c871-40ba-8322-18c95a7895e8 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:49.931986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a034a73b-29ca-44c4-89d8-a041a337a79c None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:49.940254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 205/409] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:14:49 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:49.941850 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19da74a-c871-40ba-8322-18c95a7895e8 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:49.992423 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2a0cc2-300d-42ee-97d5-d4352120846f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:49.993577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2a0cc2-300d-42ee-97d5-d4352120846f None admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:49.994076 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2a0cc2-300d-42ee-97d5-d4352120846f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:49.994544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2a0cc2-300d-42ee-97d5-d4352120846f None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85/groups/b657181658cf47efb236289d021c0f16/roles/e114d28e2dd147fcb39452dd461791c0` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:49.998605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2a0cc2-300d-42ee-97d5-d4352120846f None admin] RBAC: Authorizing `identity:create_grant(project_id=957c97404bb9448e9d03eee0d3225b85, group_id=b657181658cf47efb236289d021c0f16, role_id=e114d28e2dd147fcb39452dd461791c0)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:50.001465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2a0cc2-300d-42ee-97d5-d4352120846f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:50.032353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 205/410] 10.222.0.64 () {58 vars in 1442 bytes} [Tue Oct 4 14:14:49 2022] PUT /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85/groups/b657181658cf47efb236289d021c0f16/roles/e114d28e2dd147fcb39452dd461791c0 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:14:50.032353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2a0cc2-300d-42ee-97d5-d4352120846f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:50.101672 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:50.149267 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:50.158092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19da74a-c871-40ba-8322-18c95a7895e8 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:50.245262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf68677-5406-4911-8526-4e6fd0fe1490 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:50.246155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf68677-5406-4911-8526-4e6fd0fe1490 None admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:50.246573 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf68677-5406-4911-8526-4e6fd0fe1490 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:50.246977 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf68677-5406-4911-8526-4e6fd0fe1490 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535/groups/b657181658cf47efb236289d021c0f16/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:50.251224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf68677-5406-4911-8526-4e6fd0fe1490 None admin] RBAC: Authorizing `identity:create_grant(project_id=2728b0ff765c4e9fa2c0cb33c62b0535, group_id=b657181658cf47efb236289d021c0f16, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:50.254894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf68677-5406-4911-8526-4e6fd0fe1490 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:50.260156 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2a0cc2-300d-42ee-97d5-d4352120846f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:50.288344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 206/411] 10.222.0.64 () {58 vars in 1442 bytes} [Tue Oct 4 14:14:49 2022] PUT /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535/groups/b657181658cf47efb236289d021c0f16/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:14:50.288344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf68677-5406-4911-8526-4e6fd0fe1490 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:50.373968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1584f5-5ac8-4384-b072-69330a6c919b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:50.375773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1584f5-5ac8-4384-b072-69330a6c919b None admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:50.376420 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1584f5-5ac8-4384-b072-69330a6c919b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:50.377702 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1584f5-5ac8-4384-b072-69330a6c919b None admin] PATH_INFO: `/v3/projects/79763f634e4d4d53967547cda84e5ebd/groups/63b6321f919240fd87c9ccafb29b636c/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:50.386245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1584f5-5ac8-4384-b072-69330a6c919b None admin] RBAC: Authorizing `identity:create_grant(project_id=79763f634e4d4d53967547cda84e5ebd, group_id=63b6321f919240fd87c9ccafb29b636c, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:50.388175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1584f5-5ac8-4384-b072-69330a6c919b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:50.418079 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 206/412] 10.222.0.64 () {58 vars in 1442 bytes} [Tue Oct 4 14:14:50 2022] PUT /identity/v3/projects/79763f634e4d4d53967547cda84e5ebd/groups/63b6321f919240fd87c9ccafb29b636c/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:14:50.532532 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf68677-5406-4911-8526-4e6fd0fe1490 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:50.599109 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a86b82-d3fe-47b8-b0e3-63a777968f15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:50.600854 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a86b82-d3fe-47b8-b0e3-63a777968f15 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:50.601703 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a86b82-d3fe-47b8-b0e3-63a777968f15 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:50.602470 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a86b82-d3fe-47b8-b0e3-63a777968f15 None admin] PATH_INFO: `/v3/groups/b657181658cf47efb236289d021c0f16` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:50.605278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a86b82-d3fe-47b8-b0e3-63a777968f15 None admin] RBAC: Authorizing `identity:get_group(group_id=b657181658cf47efb236289d021c0f16)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:50.608888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a86b82-d3fe-47b8-b0e3-63a777968f15 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:50.614150 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 207/413] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Oct 4 14:14:50 2022] GET /identity/v3/groups/b657181658cf47efb236289d021c0f16 => generated 226 bytes in 332 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:50.622413 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1584f5-5ac8-4384-b072-69330a6c919b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:50.641559 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f004541b-cb61-4b90-afb0-402191977884 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:50.642496 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f004541b-cb61-4b90-afb0-402191977884 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:50.643390 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f004541b-cb61-4b90-afb0-402191977884 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:50.646954 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 208/414] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:50 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:50.653714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e13e6b-9a08-43fb-969d-c4b9e5a0251f None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:50.654638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e13e6b-9a08-43fb-969d-c4b9e5a0251f None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:50.655411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e13e6b-9a08-43fb-969d-c4b9e5a0251f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:50.694064 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-f0e13e6b-9a08-43fb-969d-c4b9e5a0251f None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:50.698149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-f0e13e6b-9a08-43fb-969d-c4b9e5a0251f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:50.747811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:50.828718 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:50.842294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1584f5-5ac8-4384-b072-69330a6c919b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:50.852505 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 209/415] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:50.918284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3e93ad-712d-45b3-a06c-db55f8b23323 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:50.919804 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3e93ad-712d-45b3-a06c-db55f8b23323 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:50.920582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3e93ad-712d-45b3-a06c-db55f8b23323 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:50.921398 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3e93ad-712d-45b3-a06c-db55f8b23323 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:50.925747 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3e93ad-712d-45b3-a06c-db55f8b23323 None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:50.928861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3e93ad-712d-45b3-a06c-db55f8b23323 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:50.932922 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 207/416] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:50 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:50.940600 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e13e6b-9a08-43fb-969d-c4b9e5a0251f None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:51.148888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e13e6b-9a08-43fb-969d-c4b9e5a0251f None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:51.224984 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a41083-34f5-425f-8675-1c39068c2b25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:51.226075 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a41083-34f5-425f-8675-1c39068c2b25 None admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:51.226653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a41083-34f5-425f-8675-1c39068c2b25 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:51.227121 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a41083-34f5-425f-8675-1c39068c2b25 None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535/groups/b657181658cf47efb236289d021c0f16/roles/e114d28e2dd147fcb39452dd461791c0` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:51.233966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a41083-34f5-425f-8675-1c39068c2b25 None admin] RBAC: Authorizing `identity:create_grant(project_id=2728b0ff765c4e9fa2c0cb33c62b0535, group_id=b657181658cf47efb236289d021c0f16, role_id=e114d28e2dd147fcb39452dd461791c0)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:51.235799 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a41083-34f5-425f-8675-1c39068c2b25 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:51.261880 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 210/417] 10.222.0.64 () {58 vars in 1442 bytes} [Tue Oct 4 14:14:50 2022] PUT /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535/groups/b657181658cf47efb236289d021c0f16/roles/e114d28e2dd147fcb39452dd461791c0 => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:14:51.347054 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3e93ad-712d-45b3-a06c-db55f8b23323 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:51.566082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3e93ad-712d-45b3-a06c-db55f8b23323 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:51.644711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05225cd1-f22b-4a7b-a9f8-e5acd25cc3a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:51.645839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05225cd1-f22b-4a7b-a9f8-e5acd25cc3a7 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:51.646343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05225cd1-f22b-4a7b-a9f8-e5acd25cc3a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:51.646827 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05225cd1-f22b-4a7b-a9f8-e5acd25cc3a7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:51.666151 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05225cd1-f22b-4a7b-a9f8-e5acd25cc3a7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:51.676078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-05225cd1-f22b-4a7b-a9f8-e5acd25cc3a7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:51.676078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:51.676078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:51.676078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:51.676078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:51.676078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:51.676078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:51.676078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:51.676078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:51.676078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:51.676078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:51.676078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:51.676078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:51.676078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:51.676078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:51.676078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:51.677097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:51.677097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:51.677097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:51.677097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:51.677097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:51.677097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:51.677097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:51.677097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:51.677097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:51.677097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:51.677097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:51.677097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:51.677097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:51.677097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:51.677097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:51.677097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:51.677899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:51.677899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:51.677899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:51.677899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:51.677899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:51.677899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:51.677899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:51.677899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:51.677899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:51.677899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:51.677899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:51.677899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:51.678509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 208/418] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 332 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:51.687943 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a41083-34f5-425f-8675-1c39068c2b25 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:51.741451 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:51.932292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a41083-34f5-425f-8675-1c39068c2b25 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:52.003956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e414fb41-f91d-4a82-982a-26d7bba3f7aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:52.005067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e414fb41-f91d-4a82-982a-26d7bba3f7aa None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:52.005557 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e414fb41-f91d-4a82-982a-26d7bba3f7aa None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:52.006089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e414fb41-f91d-4a82-982a-26d7bba3f7aa None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:52.007336 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e414fb41-f91d-4a82-982a-26d7bba3f7aa None admin] RBAC: Adding query filter params (name=member) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:52.007880 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e414fb41-f91d-4a82-982a-26d7bba3f7aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:52.010543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e414fb41-f91d-4a82-982a-26d7bba3f7aa None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:52.034359 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 211/419] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 342 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:52.040073 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05225cd1-f22b-4a7b-a9f8-e5acd25cc3a7 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:52.088690 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:52.271109 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05225cd1-f22b-4a7b-a9f8-e5acd25cc3a7 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:52.345523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715d7bc1-89f0-4b48-9c0b-b34fe73db754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:52.346716 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715d7bc1-89f0-4b48-9c0b-b34fe73db754 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:52.347157 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715d7bc1-89f0-4b48-9c0b-b34fe73db754 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:52.347626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715d7bc1-89f0-4b48-9c0b-b34fe73db754 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:52.541425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715d7bc1-89f0-4b48-9c0b-b34fe73db754 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:52.544068 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715d7bc1-89f0-4b48-9c0b-b34fe73db754 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:52.605785 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:52.805464 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 209/420] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 766 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:14:52.814591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e414fb41-f91d-4a82-982a-26d7bba3f7aa None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:52.969737 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:53.004252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e414fb41-f91d-4a82-982a-26d7bba3f7aa None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:53.066828 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d408eb22-081e-43d9-bfaa-5d084106938d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:53.067412 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d408eb22-081e-43d9-bfaa-5d084106938d None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:53.067700 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d408eb22-081e-43d9-bfaa-5d084106938d None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:53.067982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d408eb22-081e-43d9-bfaa-5d084106938d None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:53.073452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d408eb22-081e-43d9-bfaa-5d084106938d None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:53.075039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d408eb22-081e-43d9-bfaa-5d084106938d None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:53.077711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 212/421] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:52 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:53.098870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715d7bc1-89f0-4b48-9c0b-b34fe73db754 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:53.375676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715d7bc1-89f0-4b48-9c0b-b34fe73db754 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:53.439194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:53.476869 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd86306-0f23-4fb1-8f32-b09a32f732c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:53.477960 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd86306-0f23-4fb1-8f32-b09a32f732c1 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:53.478438 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd86306-0f23-4fb1-8f32-b09a32f732c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:53.478904 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd86306-0f23-4fb1-8f32-b09a32f732c1 None admin] PATH_INFO: `/v3/groups/b657181658cf47efb236289d021c0f16` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:53.485816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd86306-0f23-4fb1-8f32-b09a32f732c1 None admin] RBAC: Authorizing `identity:get_group(group_id=b657181658cf47efb236289d021c0f16)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:53.489121 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd86306-0f23-4fb1-8f32-b09a32f732c1 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:53.492675 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 210/422] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Oct 4 14:14:53 2022] GET /identity/v3/groups/b657181658cf47efb236289d021c0f16 => generated 226 bytes in 395 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:53.498553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d408eb22-081e-43d9-bfaa-5d084106938d None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:53.539568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059f6952-4168-4c7f-8cbb-40ec7802602c None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:53.540256 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059f6952-4168-4c7f-8cbb-40ec7802602c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:53.540724 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059f6952-4168-4c7f-8cbb-40ec7802602c None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:53.543544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 211/423] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:53 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:53.555711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3a48b9-3df6-432a-9fc3-c799b2c35b6f None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:53.556159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3a48b9-3df6-432a-9fc3-c799b2c35b6f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:53.557197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3a48b9-3df6-432a-9fc3-c799b2c35b6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:53.617910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-eb3a48b9-3df6-432a-9fc3-c799b2c35b6f None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:53.619925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-eb3a48b9-3df6-432a-9fc3-c799b2c35b6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:53.764487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 212/424] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:14:53.770413 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d408eb22-081e-43d9-bfaa-5d084106938d None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:53.835565 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63817ed1-6660-4541-a481-822e3279d846 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:53.836273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63817ed1-6660-4541-a481-822e3279d846 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:53.836560 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63817ed1-6660-4541-a481-822e3279d846 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:53.836843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63817ed1-6660-4541-a481-822e3279d846 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:53.837636 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63817ed1-6660-4541-a481-822e3279d846 None admin] RBAC: Adding query filter params (group.id=b657181658cf47efb236289d021c0f16, role.id=bc6aeaa495ca4f3dbe72905c744c2233, scope.project.id=957c97404bb9448e9d03eee0d3225b85) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:53.837953 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63817ed1-6660-4541-a481-822e3279d846 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:53.839547 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63817ed1-6660-4541-a481-822e3279d846 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:53.852536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 213/425] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Oct 4 14:14:53 2022] GET /identity/v3/role_assignments?group.id=b657181658cf47efb236289d021c0f16&scope.project.id=957c97404bb9448e9d03eee0d3225b85&role.id=bc6aeaa495ca4f3dbe72905c744c2233 => generated 611 bytes in 349 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:53.964365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9df4fe-4c32-45c5-b168-cb2139f5561d None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:53.965112 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9df4fe-4c32-45c5-b168-cb2139f5561d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:53.965112 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9df4fe-4c32-45c5-b168-cb2139f5561d None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:53.966712 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 213/426] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:53.973832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55dae38-885b-4efd-b8af-ff6809331539 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:53.974773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55dae38-885b-4efd-b8af-ff6809331539 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:53.974773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55dae38-885b-4efd-b8af-ff6809331539 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:54.008069 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-d55dae38-885b-4efd-b8af-ff6809331539 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:54.010080 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d55dae38-885b-4efd-b8af-ff6809331539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:54.021360 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:54.129770 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 214/427] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:54.185178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b079dd75-a578-461d-88e1-d4cb5c38a5d1 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:54.185722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b079dd75-a578-461d-88e1-d4cb5c38a5d1 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:54.186111 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b079dd75-a578-461d-88e1-d4cb5c38a5d1 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:54.189046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 214/428] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:54 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:14:54.193167 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55dae38-885b-4efd-b8af-ff6809331539 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:54.198384 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6772661a-1801-47d3-b9cf-b70a32b23d86 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:54.198876 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6772661a-1801-47d3-b9cf-b70a32b23d86 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:54.199360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6772661a-1801-47d3-b9cf-b70a32b23d86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:54.239696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-6772661a-1801-47d3-b9cf-b70a32b23d86 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:54.244754 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6772661a-1801-47d3-b9cf-b70a32b23d86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:54.371951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55dae38-885b-4efd-b8af-ff6809331539 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:54.394150 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 215/429] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:54.434154 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ab14ec-dadb-4659-a729-e398ea54284f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:54.434964 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ab14ec-dadb-4659-a729-e398ea54284f None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:54.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ab14ec-dadb-4659-a729-e398ea54284f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:54.435571 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ab14ec-dadb-4659-a729-e398ea54284f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:54.449524 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ab14ec-dadb-4659-a729-e398ea54284f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:54.454433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-24ab14ec-dadb-4659-a729-e398ea54284f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:54.454433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:54.454433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:54.454433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:54.454433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:54.454433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:54.454433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:54.454433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:54.454433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:54.454433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:54.454433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:54.454433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:54.454433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:54.454433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:54.454433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:54.454433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:54.455448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:54.455448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:54.455448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:54.455448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:54.455448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:54.455448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:54.455448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:54.455448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:54.455448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:54.455448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:54.455448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:54.455448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:54.455448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:54.455448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:54.455448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:54.455448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:54.456256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:54.456256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:54.456256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:54.456256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:54.456256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:54.456256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:54.456256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:54.456256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:54.456256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:54.456256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:54.456256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:54.456256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:54.456256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 215/430] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:14:54 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:14:54.463712 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6772661a-1801-47d3-b9cf-b70a32b23d86 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:54.530186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:54.601419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6772661a-1801-47d3-b9cf-b70a32b23d86 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:54.659238 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ab14ec-dadb-4659-a729-e398ea54284f None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:54.659679 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bda3e3b-02b9-4f61-8e88-a8ca6b5ca18d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:54.660348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bda3e3b-02b9-4f61-8e88-a8ca6b5ca18d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:54.660614 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bda3e3b-02b9-4f61-8e88-a8ca6b5ca18d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:54.660888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bda3e3b-02b9-4f61-8e88-a8ca6b5ca18d None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:54.661585 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bda3e3b-02b9-4f61-8e88-a8ca6b5ca18d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:54.661894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bda3e3b-02b9-4f61-8e88-a8ca6b5ca18d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:54.663296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bda3e3b-02b9-4f61-8e88-a8ca6b5ca18d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:54.667396 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:54.674700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 216/431] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:14:54 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:14:54.681774 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bda3e3b-02b9-4f61-8e88-a8ca6b5ca18d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:54.849416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ab14ec-dadb-4659-a729-e398ea54284f None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:54.866741 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bda3e3b-02b9-4f61-8e88-a8ca6b5ca18d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:54.896916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:54.899903 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb464556-eaf7-4d9c-9841-927022946bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:54.900250 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb464556-eaf7-4d9c-9841-927022946bbb None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:54.900517 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb464556-eaf7-4d9c-9841-927022946bbb None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:54.900880 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb464556-eaf7-4d9c-9841-927022946bbb None admin] PATH_INFO: `/v3/roles/member` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:54.915818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb464556-eaf7-4d9c-9841-927022946bbb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:54.920682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-fb464556-eaf7-4d9c-9841-927022946bbb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:54.920682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:54.920682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:54.920682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:54.920682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:54.920682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:54.920682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:54.920682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:54.920682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:54.920682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:54.920682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:54.920682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:54.920682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:54.920682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:54.920682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:54.920682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:54.924847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:54.924847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:54.924847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:54.924847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:54.924847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:54.924847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:54.924847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:54.924847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:54.924847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:54.924847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:54.924847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:54.924847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:54.924847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:54.924847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:54.924847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:54.924847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:54.925686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:54.925686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:54.925686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:54.925686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:54.925686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:54.925686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:54.925686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:54.925686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:54.925686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:54.925686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:54.925686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 14:14:54.925686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:54.925686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 216/432] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:54.925686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb464556-eaf7-4d9c-9841-927022946bbb None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:54.945796 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947f2ce1-3cff-460d-9bc2-f12d8552e78c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:54.946762 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947f2ce1-3cff-460d-9bc2-f12d8552e78c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:54.947212 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947f2ce1-3cff-460d-9bc2-f12d8552e78c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:54.947707 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947f2ce1-3cff-460d-9bc2-f12d8552e78c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:55.007474 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:55.051649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947f2ce1-3cff-460d-9bc2-f12d8552e78c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:55.053203 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947f2ce1-3cff-460d-9bc2-f12d8552e78c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:55.063218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb464556-eaf7-4d9c-9841-927022946bbb None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:55.104189 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87379576-d1be-47ba-b163-4b85f7ff09d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:55.104866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87379576-d1be-47ba-b163-4b85f7ff09d6 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:55.105150 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87379576-d1be-47ba-b163-4b85f7ff09d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:55.105424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87379576-d1be-47ba-b163-4b85f7ff09d6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:55.117995 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87379576-d1be-47ba-b163-4b85f7ff09d6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:55.122988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-87379576-d1be-47ba-b163-4b85f7ff09d6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:14:55.122988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:55.122988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:55.122988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:55.122988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:55.122988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:55.122988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:55.122988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:55.122988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:55.122988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:55.122988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:55.122988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:55.122988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:55.122988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:55.122988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:55.122988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:55.124194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:55.124194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:55.124194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:55.124194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:55.124194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:55.124194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:55.124194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:55.124194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:55.124194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:55.124194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:55.124194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:55.124194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:55.124194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:55.124194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:55.124194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:55.124194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:55.125071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:55.125071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:55.125071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:55.125071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:55.125071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:55.125071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:55.125071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:55.125071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:55.125071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:55.125071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:55.125071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:14:55.125071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:14:55.125071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 217/433] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:14:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:55.125911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87379576-d1be-47ba-b163-4b85f7ff09d6 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:55.182954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 217/434] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 495 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 14:14:55.182954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4c7e5e-eeec-43b8-82e3-c59f3b754fa8 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:55.182954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4c7e5e-eeec-43b8-82e3-c59f3b754fa8 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:55.182954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4c7e5e-eeec-43b8-82e3-c59f3b754fa8 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:55.182954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 218/435] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:14:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:14:55.182954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4c7e5e-eeec-43b8-82e3-c59f3b754fa8 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:55.206067 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:55.257981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87379576-d1be-47ba-b163-4b85f7ff09d6 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:55.284264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:55.300343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1552757a-3dc4-415b-be09-1528f354bfce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:55.300989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1552757a-3dc4-415b-be09-1528f354bfce None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:55.301368 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1552757a-3dc4-415b-be09-1528f354bfce None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:55.301649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1552757a-3dc4-415b-be09-1528f354bfce None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:55.302431 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1552757a-3dc4-415b-be09-1528f354bfce None admin] RBAC: Adding query filter params (name=member) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:55.302747 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1552757a-3dc4-415b-be09-1528f354bfce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:55.304352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1552757a-3dc4-415b-be09-1528f354bfce None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:55.312252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4c7e5e-eeec-43b8-82e3-c59f3b754fa8 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:55.315299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 218/436] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:14:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:55.316382 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1552757a-3dc4-415b-be09-1528f354bfce None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:55.367214 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905d4fde-37ef-4adb-acc7-43aaf40207ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:55.368195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905d4fde-37ef-4adb-acc7-43aaf40207ae None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:55.368679 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905d4fde-37ef-4adb-acc7-43aaf40207ae None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:55.369118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905d4fde-37ef-4adb-acc7-43aaf40207ae None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:55.370718 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905d4fde-37ef-4adb-acc7-43aaf40207ae None admin] RBAC: Adding query filter params (name=admin) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:55.371223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905d4fde-37ef-4adb-acc7-43aaf40207ae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:55.373296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905d4fde-37ef-4adb-acc7-43aaf40207ae None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:55.388046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 219/437] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:14:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:55.390229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25526bf6-ea90-4aef-b089-abea131da820 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:55.390709 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25526bf6-ea90-4aef-b089-abea131da820 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:55.391180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25526bf6-ea90-4aef-b089-abea131da820 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:55.415951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-25526bf6-ea90-4aef-b089-abea131da820 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:14:55.417823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-25526bf6-ea90-4aef-b089-abea131da820 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:14:55.478864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1552757a-3dc4-415b-be09-1528f354bfce None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:55.519735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 220/438] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:14:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:14:55.520062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629fe653-012b-4955-97d8-7b72e5b29f39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:55.520830 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629fe653-012b-4955-97d8-7b72e5b29f39 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:55.520987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25526bf6-ea90-4aef-b089-abea131da820 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:55.521120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629fe653-012b-4955-97d8-7b72e5b29f39 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:55.521449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629fe653-012b-4955-97d8-7b72e5b29f39 None admin] PATH_INFO: `/v3/projects/957c97404bb9448e9d03eee0d3225b85` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:55.522766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629fe653-012b-4955-97d8-7b72e5b29f39 None admin] RBAC: Authorizing `identity:get_project(project_id=957c97404bb9448e9d03eee0d3225b85)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:55.524240 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629fe653-012b-4955-97d8-7b72e5b29f39 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:55.526277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 219/439] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:55 2022] GET /identity/v3/projects/957c97404bb9448e9d03eee0d3225b85 => generated 298 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:55.526825 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629fe653-012b-4955-97d8-7b72e5b29f39 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:55.527956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:55.597099 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:55.663324 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629fe653-012b-4955-97d8-7b72e5b29f39 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:55.670271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25526bf6-ea90-4aef-b089-abea131da820 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:55.710702 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12be425b-d11c-4a46-8e71-1e3e87be868f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:55.711224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12be425b-d11c-4a46-8e71-1e3e87be868f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:55.711506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12be425b-d11c-4a46-8e71-1e3e87be868f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:55.711780 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12be425b-d11c-4a46-8e71-1e3e87be868f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:55.724989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d3826d-b268-4da7-9704-de5560477ef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:55.726064 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d3826d-b268-4da7-9704-de5560477ef9 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:55.726506 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d3826d-b268-4da7-9704-de5560477ef9 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:55.726918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d3826d-b268-4da7-9704-de5560477ef9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:55.791898 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12be425b-d11c-4a46-8e71-1e3e87be868f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:55.793420 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12be425b-d11c-4a46-8e71-1e3e87be868f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:55.796710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:55.841338 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d3826d-b268-4da7-9704-de5560477ef9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:55.842813 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d3826d-b268-4da7-9704-de5560477ef9 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:55.910881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:55.932998 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 221/440] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:14:55.932998 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12be425b-d11c-4a46-8e71-1e3e87be868f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:55.977588 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 220/441] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:14:55.978206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d3826d-b268-4da7-9704-de5560477ef9 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:56.097167 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12be425b-d11c-4a46-8e71-1e3e87be868f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:56.097954 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d3826d-b268-4da7-9704-de5560477ef9 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:56.101725 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:56.140350 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e7251d-cf36-44d2-9657-6fdbacd45c4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:56.140999 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e7251d-cf36-44d2-9657-6fdbacd45c4d None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:56.141281 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e7251d-cf36-44d2-9657-6fdbacd45c4d None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:56.141557 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e7251d-cf36-44d2-9657-6fdbacd45c4d None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:56.143521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e7251d-cf36-44d2-9657-6fdbacd45c4d None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:56.144827 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12c62c5-7ba3-484d-85ae-39888ac9bce3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:56.145296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e7251d-cf36-44d2-9657-6fdbacd45c4d None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:56.145597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12c62c5-7ba3-484d-85ae-39888ac9bce3 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:56.145953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12c62c5-7ba3-484d-85ae-39888ac9bce3 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:56.146288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12c62c5-7ba3-484d-85ae-39888ac9bce3 None admin] PATH_INFO: `/v3/groups/b657181658cf47efb236289d021c0f16` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:56.147042 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 221/442] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:55 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:56.147841 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e7251d-cf36-44d2-9657-6fdbacd45c4d None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:56.148021 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12c62c5-7ba3-484d-85ae-39888ac9bce3 None admin] RBAC: Authorizing `identity:get_group(group_id=b657181658cf47efb236289d021c0f16)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:56.150460 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12c62c5-7ba3-484d-85ae-39888ac9bce3 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:56.153445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 222/443] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Oct 4 14:14:55 2022] GET /identity/v3/groups/b657181658cf47efb236289d021c0f16 => generated 226 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:56.154537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12c62c5-7ba3-484d-85ae-39888ac9bce3 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:56.178371 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:56.286334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12c62c5-7ba3-484d-85ae-39888ac9bce3 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:56.307962 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e7251d-cf36-44d2-9657-6fdbacd45c4d None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:56.340459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869654dd-47f8-4eed-b0d8-af8c4fa1854f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:56.341471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869654dd-47f8-4eed-b0d8-af8c4fa1854f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:56.341953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869654dd-47f8-4eed-b0d8-af8c4fa1854f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:56.342411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869654dd-47f8-4eed-b0d8-af8c4fa1854f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:56.349493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f973e8c-d897-4ba0-8355-86cd8f594942 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:56.350135 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f973e8c-d897-4ba0-8355-86cd8f594942 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:56.350416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f973e8c-d897-4ba0-8355-86cd8f594942 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:56.350691 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f973e8c-d897-4ba0-8355-86cd8f594942 None admin] PATH_INFO: `/v3/projects/79763f634e4d4d53967547cda84e5ebd` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:56.351894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f973e8c-d897-4ba0-8355-86cd8f594942 None admin] RBAC: Authorizing `identity:get_project(project_id=79763f634e4d4d53967547cda84e5ebd)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:56.353532 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f973e8c-d897-4ba0-8355-86cd8f594942 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:56.355352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 222/444] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:56 2022] GET /identity/v3/projects/79763f634e4d4d53967547cda84e5ebd => generated 344 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:56.356223 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f973e8c-d897-4ba0-8355-86cd8f594942 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:56.369204 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869654dd-47f8-4eed-b0d8-af8c4fa1854f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:14:56.379393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-869654dd-47f8-4eed-b0d8-af8c4fa1854f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:56.379393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:14:56.379393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:14:56.379393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:14:56.379393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:14:56.379393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:14:56.379393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:14:56.379393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:14:56.379393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:14:56.379393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:14:56.379393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:14:56.379393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:14:56.379393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:14:56.379393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:14:56.379393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:14:56.379393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:14:56.381195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:14:56.381195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:14:56.381195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:14:56.381195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:14:56.381195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:14:56.381195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:14:56.381195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:14:56.381195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:14:56.381195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:14:56.381195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:14:56.381195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:14:56.381195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:14:56.381195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:14:56.381195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:14:56.381195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:14:56.381195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:14:56.382568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:14:56.382568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:14:56.382568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:14:56.382568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:14:56.382568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:14:56.382568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:14:56.382568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:14:56.382568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:14:56.382568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:14:56.382568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:14:56.382568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:14:56.382568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:14:56.382568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 223/445] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:14:56 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:14:56.383723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869654dd-47f8-4eed-b0d8-af8c4fa1854f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:56.444205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:56.485393 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:56.506135 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f973e8c-d897-4ba0-8355-86cd8f594942 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:56.551494 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869654dd-47f8-4eed-b0d8-af8c4fa1854f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:56.554030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53de9614-49c4-4645-8ddc-a69c493aeef4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:56.554792 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53de9614-49c4-4645-8ddc-a69c493aeef4 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:56.555210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53de9614-49c4-4645-8ddc-a69c493aeef4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:56.555602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53de9614-49c4-4645-8ddc-a69c493aeef4 None admin] PATH_INFO: `/v3/groups/b657181658cf47efb236289d021c0f16` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:56.557048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53de9614-49c4-4645-8ddc-a69c493aeef4 None admin] RBAC: Authorizing `identity:get_group(group_id=b657181658cf47efb236289d021c0f16)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:56.558673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53de9614-49c4-4645-8ddc-a69c493aeef4 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:56.560925 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 223/446] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Oct 4 14:14:56 2022] GET /identity/v3/groups/b657181658cf47efb236289d021c0f16 => generated 226 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:56.561834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53de9614-49c4-4645-8ddc-a69c493aeef4 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:56.603589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580e9e7d-98e1-4c37-87e0-efdd5b4b5954 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:56.604405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580e9e7d-98e1-4c37-87e0-efdd5b4b5954 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:56.604818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580e9e7d-98e1-4c37-87e0-efdd5b4b5954 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:56.605220 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580e9e7d-98e1-4c37-87e0-efdd5b4b5954 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:56.606266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580e9e7d-98e1-4c37-87e0-efdd5b4b5954 None admin] RBAC: Adding query filter params (group.id=b657181658cf47efb236289d021c0f16, role.id=e114d28e2dd147fcb39452dd461791c0, scope.project.id=957c97404bb9448e9d03eee0d3225b85) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:56.606655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580e9e7d-98e1-4c37-87e0-efdd5b4b5954 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:56.608342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580e9e7d-98e1-4c37-87e0-efdd5b4b5954 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:56.617336 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 224/447] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Oct 4 14:14:56 2022] GET /identity/v3/role_assignments?group.id=b657181658cf47efb236289d021c0f16&scope.project.id=957c97404bb9448e9d03eee0d3225b85&role.id=e114d28e2dd147fcb39452dd461791c0 => generated 611 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:56.618735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580e9e7d-98e1-4c37-87e0-efdd5b4b5954 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:56.735839 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53de9614-49c4-4645-8ddc-a69c493aeef4 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:56.774274 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:56.786546 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580e9e7d-98e1-4c37-87e0-efdd5b4b5954 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:56.801042 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33197a3c-738f-46ca-8402-cee74253e1ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:56.802186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33197a3c-738f-46ca-8402-cee74253e1ac None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:56.802621 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33197a3c-738f-46ca-8402-cee74253e1ac None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:56.803082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33197a3c-738f-46ca-8402-cee74253e1ac None admin] PATH_INFO: `/v3/groups/63b6321f919240fd87c9ccafb29b636c` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:56.805466 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33197a3c-738f-46ca-8402-cee74253e1ac None admin] RBAC: Authorizing `identity:get_group(group_id=63b6321f919240fd87c9ccafb29b636c)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:56.808465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33197a3c-738f-46ca-8402-cee74253e1ac None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:56.820194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 224/448] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Oct 4 14:14:56 2022] GET /identity/v3/groups/63b6321f919240fd87c9ccafb29b636c => generated 229 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:56.820194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33197a3c-738f-46ca-8402-cee74253e1ac None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:56.857310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7e9bad-be4b-49a5-8026-76f1179661ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:56.858166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7e9bad-be4b-49a5-8026-76f1179661ae None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:56.859520 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7e9bad-be4b-49a5-8026-76f1179661ae None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:56.859520 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7e9bad-be4b-49a5-8026-76f1179661ae None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:56.860246 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7e9bad-be4b-49a5-8026-76f1179661ae None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:56.860756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7e9bad-be4b-49a5-8026-76f1179661ae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:56.864168 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7e9bad-be4b-49a5-8026-76f1179661ae None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:56.866895 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:56.878526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 225/449] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:14:56 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:56.879699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7e9bad-be4b-49a5-8026-76f1179661ae None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:57.016666 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33197a3c-738f-46ca-8402-cee74253e1ac None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:57.061148 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7e9bad-be4b-49a5-8026-76f1179661ae None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:57.073597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b1fa79-caf6-4def-82f7-df7e34e959d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:57.074586 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b1fa79-caf6-4def-82f7-df7e34e959d3 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:57.075016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b1fa79-caf6-4def-82f7-df7e34e959d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:57.075480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b1fa79-caf6-4def-82f7-df7e34e959d3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:57.076796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b1fa79-caf6-4def-82f7-df7e34e959d3 None admin] RBAC: Adding query filter params (group.id=b657181658cf47efb236289d021c0f16, role.id=bc6aeaa495ca4f3dbe72905c744c2233, scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:57.077326 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b1fa79-caf6-4def-82f7-df7e34e959d3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:57.080506 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b1fa79-caf6-4def-82f7-df7e34e959d3 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:57.089805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 225/450] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Oct 4 14:14:56 2022] GET /identity/v3/role_assignments?group.id=b657181658cf47efb236289d021c0f16&scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535&role.id=bc6aeaa495ca4f3dbe72905c744c2233 => generated 611 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:57.090982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b1fa79-caf6-4def-82f7-df7e34e959d3 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:57.132924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c376ca-9e79-4ec4-9b99-bde63cb8ab79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:57.133876 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c376ca-9e79-4ec4-9b99-bde63cb8ab79 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:57.134293 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c376ca-9e79-4ec4-9b99-bde63cb8ab79 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:57.134695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c376ca-9e79-4ec4-9b99-bde63cb8ab79 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:57.136997 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c376ca-9e79-4ec4-9b99-bde63cb8ab79 None admin] RBAC: Adding query filter params (group.id=63b6321f919240fd87c9ccafb29b636c, role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=79763f634e4d4d53967547cda84e5ebd) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:57.137463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c376ca-9e79-4ec4-9b99-bde63cb8ab79 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:57.140205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c376ca-9e79-4ec4-9b99-bde63cb8ab79 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:57.155547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 226/451] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Oct 4 14:14:56 2022] GET /identity/v3/role_assignments?group.id=63b6321f919240fd87c9ccafb29b636c&scope.project.id=79763f634e4d4d53967547cda84e5ebd&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 611 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:57.285501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:57.311953 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b1fa79-caf6-4def-82f7-df7e34e959d3 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:57.385715 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b18220-3a6e-43aa-93ae-066a01dafc18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:57.386436 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b18220-3a6e-43aa-93ae-066a01dafc18 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:57.386774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b18220-3a6e-43aa-93ae-066a01dafc18 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:57.387105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b18220-3a6e-43aa-93ae-066a01dafc18 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:57.514980 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b18220-3a6e-43aa-93ae-066a01dafc18 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:57.516564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b18220-3a6e-43aa-93ae-066a01dafc18 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:57.645457 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:57.662118 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 226/452] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:14:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 572 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:14:57.668582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c376ca-9e79-4ec4-9b99-bde63cb8ab79 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:57.712706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:57.861356 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c376ca-9e79-4ec4-9b99-bde63cb8ab79 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:57.921155 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91b83f3-66f3-4c3a-ba71-530ffc07428f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:57.921798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91b83f3-66f3-4c3a-ba71-530ffc07428f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:57.922166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91b83f3-66f3-4c3a-ba71-530ffc07428f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:57.922522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91b83f3-66f3-4c3a-ba71-530ffc07428f None admin] PATH_INFO: `/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:57.924064 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91b83f3-66f3-4c3a-ba71-530ffc07428f None admin] RBAC: Authorizing `identity:get_project(project_id=2728b0ff765c4e9fa2c0cb33c62b0535)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:57.925652 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91b83f3-66f3-4c3a-ba71-530ffc07428f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:57.927565 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 227/453] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:14:57 2022] GET /identity/v3/projects/2728b0ff765c4e9fa2c0cb33c62b0535 => generated 302 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:57.933711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b18220-3a6e-43aa-93ae-066a01dafc18 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:58.054757 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b18220-3a6e-43aa-93ae-066a01dafc18 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:58.099420 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8daddd50-37b6-44f6-a549-9d6233b9b769 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:58.100201 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daddd50-37b6-44f6-a549-9d6233b9b769 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:58.100616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daddd50-37b6-44f6-a549-9d6233b9b769 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:58.100981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daddd50-37b6-44f6-a549-9d6233b9b769 None admin] PATH_INFO: `/v3/groups/b657181658cf47efb236289d021c0f16` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:58.102403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8daddd50-37b6-44f6-a549-9d6233b9b769 None admin] RBAC: Authorizing `identity:get_group(group_id=b657181658cf47efb236289d021c0f16)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:58.104025 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8daddd50-37b6-44f6-a549-9d6233b9b769 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:58.106211 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 227/454] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Oct 4 14:14:57 2022] GET /identity/v3/groups/b657181658cf47efb236289d021c0f16 => generated 226 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:14:58.113304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91b83f3-66f3-4c3a-ba71-530ffc07428f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:14:58.226531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:14:58.237758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91b83f3-66f3-4c3a-ba71-530ffc07428f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:14:58.277898 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6ad7f6-81ae-46e5-933c-5773636e0457 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:14:58.278566 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6ad7f6-81ae-46e5-933c-5773636e0457 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:14:58.278886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6ad7f6-81ae-46e5-933c-5773636e0457 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:14:58.279210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6ad7f6-81ae-46e5-933c-5773636e0457 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:14:58.280004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6ad7f6-81ae-46e5-933c-5773636e0457 None admin] RBAC: Adding query filter params (group.id=b657181658cf47efb236289d021c0f16, role.id=e114d28e2dd147fcb39452dd461791c0, scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:14:58.280382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6ad7f6-81ae-46e5-933c-5773636e0457 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:14:58.282237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6ad7f6-81ae-46e5-933c-5773636e0457 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:14:58.289470 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 228/455] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Oct 4 14:14:58 2022] GET /identity/v3/role_assignments?group.id=b657181658cf47efb236289d021c0f16&scope.project.id=2728b0ff765c4e9fa2c0cb33c62b0535&role.id=e114d28e2dd147fcb39452dd461791c0 => generated 611 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:15:01.341033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7864eaa-9d19-4416-aec8-591c7f4a5931 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:01.341569 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7864eaa-9d19-4416-aec8-591c7f4a5931 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:01.341569 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7864eaa-9d19-4416-aec8-591c7f4a5931 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:01.343298 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 228/456] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:01.354206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f034c1d-f40a-488a-9df4-42c9cae7d100 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:01.354753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f034c1d-f40a-488a-9df4-42c9cae7d100 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:01.354753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f034c1d-f40a-488a-9df4-42c9cae7d100 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:01.389936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-6f034c1d-f40a-488a-9df4-42c9cae7d100 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:01.391161 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6f034c1d-f40a-488a-9df4-42c9cae7d100 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:01.495255 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 229/457] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:01.742352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903e1504-29aa-4367-85cc-99896b0178ac None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:01.743352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903e1504-29aa-4367-85cc-99896b0178ac None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:01.744175 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903e1504-29aa-4367-85cc-99896b0178ac None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:01.747348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 229/458] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:01 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:01.755493 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7066bb86-03f4-4805-bf14-413bf333384c None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:01.755764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7066bb86-03f4-4805-bf14-413bf333384c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:01.756243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7066bb86-03f4-4805-bf14-413bf333384c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:01.783748 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-7066bb86-03f4-4805-bf14-413bf333384c None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:01.785462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7066bb86-03f4-4805-bf14-413bf333384c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:01.908982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 230/459] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:15:02.056264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903e1504-29aa-4367-85cc-99896b0178ac None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:02.146360 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:02.217625 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903e1504-29aa-4367-85cc-99896b0178ac None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:02.267612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78cf7d0-ac9b-4884-a615-27d23e91877d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:02.268938 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78cf7d0-ac9b-4884-a615-27d23e91877d None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:02.269444 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78cf7d0-ac9b-4884-a615-27d23e91877d None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:02.269915 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78cf7d0-ac9b-4884-a615-27d23e91877d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:02.292703 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78cf7d0-ac9b-4884-a615-27d23e91877d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:02.294574 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78cf7d0-ac9b-4884-a615-27d23e91877d None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:02.316304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-f78cf7d0-ac9b-4884-a615-27d23e91877d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:02.316304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:02.316304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:02.316304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:02.316304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:02.316304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:02.316304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:02.316304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:02.316304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:02.316304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:02.316304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:02.316304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:02.316304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:02.316304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:02.316304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:02.316304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:02.317297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:02.317297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:02.317297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:02.317297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:02.317297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:02.317297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:02.317297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:02.317297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:02.317297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:02.317297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:02.317297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:02.317297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:02.317297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:02.317297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:02.317297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:02.317297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:02.317297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:02.318199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:02.318199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:02.318199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:02.320230 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 230/460] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:02.330417 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7066bb86-03f4-4805-bf14-413bf333384c None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:02.356794 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:02.516821 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7066bb86-03f4-4805-bf14-413bf333384c None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:02.539551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90163964-ffe4-4093-a3ab-d283a047d299 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:02.540643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90163964-ffe4-4093-a3ab-d283a047d299 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:02.541177 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90163964-ffe4-4093-a3ab-d283a047d299 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:02.543466 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 231/461] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:15:02.544886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90163964-ffe4-4093-a3ab-d283a047d299 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:02.576860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb42c604-a030-4186-b117-233afff8e3d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:02.578085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb42c604-a030-4186-b117-233afff8e3d3 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:02.578616 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb42c604-a030-4186-b117-233afff8e3d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:02.579082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb42c604-a030-4186-b117-233afff8e3d3 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:02.580542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb42c604-a030-4186-b117-233afff8e3d3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:02.581453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb42c604-a030-4186-b117-233afff8e3d3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:02.583686 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb42c604-a030-4186-b117-233afff8e3d3 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:02.599795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 231/462] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:02.602185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d3b6b3-5b23-4cf4-b800-ff6229b75826 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:02.602913 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d3b6b3-5b23-4cf4-b800-ff6229b75826 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:02.603523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d3b6b3-5b23-4cf4-b800-ff6229b75826 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:02.629302 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-73d3b6b3-5b23-4cf4-b800-ff6229b75826 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:02.631564 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-73d3b6b3-5b23-4cf4-b800-ff6229b75826 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:02.727266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 232/463] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:02.728144 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d3b6b3-5b23-4cf4-b800-ff6229b75826 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:02.733262 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:02.788168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90163964-ffe4-4093-a3ab-d283a047d299 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:02.826527 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:02.898791 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a415b0-34d8-4def-878e-29453bea53f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:02.899446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a415b0-34d8-4def-878e-29453bea53f9 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:02.899722 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a415b0-34d8-4def-878e-29453bea53f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:02.900037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a415b0-34d8-4def-878e-29453bea53f9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:02.944482 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a415b0-34d8-4def-878e-29453bea53f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:02.945984 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a415b0-34d8-4def-878e-29453bea53f9 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:02.990597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-78a415b0-34d8-4def-878e-29453bea53f9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:02.990597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:02.990597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:02.990597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:02.990597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:02.990597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:02.990597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:02.990597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:02.990597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:02.990597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:02.990597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:02.990597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:02.990597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:02.990597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:02.990597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:02.990597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:02.992416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:02.992416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:02.992416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:02.992416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:02.992416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:02.992416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:02.992416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:02.992416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:02.992416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:02.992416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:02.992416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:02.992416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:02.992416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:02.992416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:02.992416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:02.992416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:02.992416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:02.994218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:02.994218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:02.994218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:02.995291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 232/464] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 451 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:03.004534 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a415b0-34d8-4def-878e-29453bea53f9 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:03.043381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d3b6b3-5b23-4cf4-b800-ff6229b75826 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:03.193291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7e29e3-a5dc-4a64-884c-52465deceaa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:03.194349 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7e29e3-a5dc-4a64-884c-52465deceaa5 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:03.194809 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7e29e3-a5dc-4a64-884c-52465deceaa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:03.195276 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7e29e3-a5dc-4a64-884c-52465deceaa5 None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:03.196545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7e29e3-a5dc-4a64-884c-52465deceaa5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:03.201873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7e29e3-a5dc-4a64-884c-52465deceaa5 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:03.276371 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 233/465] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:15:02 2022] POST /identity/v3/users => generated 248 bytes in 539 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:15:03.308626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7e29e3-a5dc-4a64-884c-52465deceaa5 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:03.355480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a415b0-34d8-4def-878e-29453bea53f9 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:03.400775 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:03.425609 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f04642-efae-4805-b231-0ae6adf646f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:03.426254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f04642-efae-4805-b231-0ae6adf646f0 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:03.426535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f04642-efae-4805-b231-0ae6adf646f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:03.426820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f04642-efae-4805-b231-0ae6adf646f0 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:03.427524 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f04642-efae-4805-b231-0ae6adf646f0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:03.427840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f04642-efae-4805-b231-0ae6adf646f0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:03.431332 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f04642-efae-4805-b231-0ae6adf646f0 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:03.450343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 233/466] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 447 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:03.452119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc3698e-10b8-43ac-9b2d-6bb9047f7dd1 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:03.452511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc3698e-10b8-43ac-9b2d-6bb9047f7dd1 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:03.452682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc3698e-10b8-43ac-9b2d-6bb9047f7dd1 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:03.454618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 234/467] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:03.461705 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc3698e-10b8-43ac-9b2d-6bb9047f7dd1 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:03.483477 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7e29e3-a5dc-4a64-884c-52465deceaa5 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:03.506627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:03.533430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcebf368-2de6-4a00-bf52-f7378000a6bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:03.534358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcebf368-2de6-4a00-bf52-f7378000a6bb None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:03.534788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcebf368-2de6-4a00-bf52-f7378000a6bb None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:03.535209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcebf368-2de6-4a00-bf52-f7378000a6bb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:03.548413 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcebf368-2de6-4a00-bf52-f7378000a6bb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:03.550332 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcebf368-2de6-4a00-bf52-f7378000a6bb None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:03.569337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-fcebf368-2de6-4a00-bf52-f7378000a6bb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:03.569337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:03.569337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:03.569337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:03.569337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:03.569337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:03.569337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:03.569337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:03.569337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:03.569337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:03.569337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:03.569337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:03.569337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:03.569337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:03.569337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:03.569337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:03.570325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:03.570325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:03.570325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:03.570325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:03.570325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:03.570325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:03.570325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:03.570325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:03.570325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:03.570325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:03.570325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:03.570325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:03.570325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:03.570325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:03.570325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:03.570325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:03.570325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:03.571175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:03.571175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:03.571175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:03.573503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 234/468] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:03.575102 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5cc6cd-5ed2-4dc3-8285-b952af06f181 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:03.575571 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5cc6cd-5ed2-4dc3-8285-b952af06f181 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:03.576017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5cc6cd-5ed2-4dc3-8285-b952af06f181 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:03.605684 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-bd5cc6cd-5ed2-4dc3-8285-b952af06f181 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:03.607733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-bd5cc6cd-5ed2-4dc3-8285-b952af06f181 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:03.675619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc3698e-10b8-43ac-9b2d-6bb9047f7dd1 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:03.728872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 235/469] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:03.729673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5cc6cd-5ed2-4dc3-8285-b952af06f181 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:03.745037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa440cc-668f-4b0f-8ce2-47299846f00b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:03.745655 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa440cc-668f-4b0f-8ce2-47299846f00b None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:03.745944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa440cc-668f-4b0f-8ce2-47299846f00b None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:03.746206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa440cc-668f-4b0f-8ce2-47299846f00b None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:03.746848 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa440cc-668f-4b0f-8ce2-47299846f00b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:03.748613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa440cc-668f-4b0f-8ce2-47299846f00b None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:03.776897 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 235/470] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:15:03 2022] POST /identity/v3/users => generated 248 bytes in 315 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:15:03.778706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b504e8e-fc63-41ec-88a6-a930690f3514 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:03.779045 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b504e8e-fc63-41ec-88a6-a930690f3514 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:03.779362 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b504e8e-fc63-41ec-88a6-a930690f3514 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:03.782122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 236/471] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:03.787320 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:03.789707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204f377c-c9d4-45a5-8cc5-eda07ae30bc8 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:03.790300 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204f377c-c9d4-45a5-8cc5-eda07ae30bc8 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:03.790881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204f377c-c9d4-45a5-8cc5-eda07ae30bc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:03.812810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-204f377c-c9d4-45a5-8cc5-eda07ae30bc8 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:03.814712 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-204f377c-c9d4-45a5-8cc5-eda07ae30bc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:03.872198 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5cc6cd-5ed2-4dc3-8285-b952af06f181 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:03.905434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 237/472] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:15:03.927397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a459a288-5c63-46a6-a420-b8d7f32da3a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:03.928093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a459a288-5c63-46a6-a420-b8d7f32da3a7 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:03.928412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a459a288-5c63-46a6-a420-b8d7f32da3a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:03.928685 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a459a288-5c63-46a6-a420-b8d7f32da3a7 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:03.929360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a459a288-5c63-46a6-a420-b8d7f32da3a7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:03.929708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a459a288-5c63-46a6-a420-b8d7f32da3a7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:03.931230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a459a288-5c63-46a6-a420-b8d7f32da3a7 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:03.945980 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 236/473] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:03.955930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204f377c-c9d4-45a5-8cc5-eda07ae30bc8 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:03.976344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:04.164964 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204f377c-c9d4-45a5-8cc5-eda07ae30bc8 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:04.209383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef281cf8-f90b-42a9-aaa8-a21ffaa53f3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:04.210017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef281cf8-f90b-42a9-aaa8-a21ffaa53f3f None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:04.210293 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef281cf8-f90b-42a9-aaa8-a21ffaa53f3f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:04.210561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef281cf8-f90b-42a9-aaa8-a21ffaa53f3f None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:04.211218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef281cf8-f90b-42a9-aaa8-a21ffaa53f3f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:04.212784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef281cf8-f90b-42a9-aaa8-a21ffaa53f3f None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:04.239025 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 238/474] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:15:03 2022] POST /identity/v3/users => generated 246 bytes in 284 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:15:04.242942 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a459a288-5c63-46a6-a420-b8d7f32da3a7 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:04.374176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a459a288-5c63-46a6-a420-b8d7f32da3a7 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:04.377490 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:04.427541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f9a11c-9a2b-446b-a08b-3c43b50cdd09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:04.428379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f9a11c-9a2b-446b-a08b-3c43b50cdd09 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:04.428379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f9a11c-9a2b-446b-a08b-3c43b50cdd09 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:04.428698 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f9a11c-9a2b-446b-a08b-3c43b50cdd09 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:04.446932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f9a11c-9a2b-446b-a08b-3c43b50cdd09 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:04.448563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f9a11c-9a2b-446b-a08b-3c43b50cdd09 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:04.490818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-54f9a11c-9a2b-446b-a08b-3c43b50cdd09 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:04.490818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:04.490818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:04.490818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:04.490818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:04.490818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:04.490818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:04.490818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:04.490818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:04.490818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:04.490818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:04.490818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:04.490818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:04.490818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:04.490818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:04.490818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:04.492969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:04.492969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:04.492969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:04.492969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:04.492969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:04.492969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:04.492969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:04.492969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:04.492969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:04.492969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:04.492969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:04.492969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:04.492969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:04.492969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:04.492969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:04.492969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:04.492969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:04.495028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:04.495028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:04.495028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:04.495028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 237/475] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:04.502312 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef281cf8-f90b-42a9-aaa8-a21ffaa53f3f None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:04.558666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f9a11c-9a2b-446b-a08b-3c43b50cdd09 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:04.645941 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:04.741174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef281cf8-f90b-42a9-aaa8-a21ffaa53f3f None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:04.778566 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f9a11c-9a2b-446b-a08b-3c43b50cdd09 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:04.819265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7752b389-1fa4-47eb-a363-37433e8c3d0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:04.821491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7752b389-1fa4-47eb-a363-37433e8c3d0f None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:04.821985 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7752b389-1fa4-47eb-a363-37433e8c3d0f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:04.822444 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7752b389-1fa4-47eb-a363-37433e8c3d0f None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:04.823588 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7752b389-1fa4-47eb-a363-37433e8c3d0f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:04.824101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7752b389-1fa4-47eb-a363-37433e8c3d0f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:04.827713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7752b389-1fa4-47eb-a363-37433e8c3d0f None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:04.856953 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 239/476] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 356 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:04.860028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650e8810-728a-45e8-86c1-9a210741bdbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:04.861362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650e8810-728a-45e8-86c1-9a210741bdbb None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:04.861826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650e8810-728a-45e8-86c1-9a210741bdbb None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:04.862289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650e8810-728a-45e8-86c1-9a210741bdbb None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:04.863446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650e8810-728a-45e8-86c1-9a210741bdbb None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:04.863836 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7752b389-1fa4-47eb-a363-37433e8c3d0f None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:04.866028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650e8810-728a-45e8-86c1-9a210741bdbb None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:04.882303 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.sql.core [None req-650e8810-728a-45e8-86c1-9a210741bdbb None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 04 14:15:04.882303 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 04 14:15:04.882303 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [parameters: {'id': '77dd595a7f4742a6987a00ae4f913745', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 04 14:15:04.882303 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81461) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 04 14:15:04.920219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-650e8810-728a-45e8-86c1-9a210741bdbb None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 04 14:15:04.920219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:04.920219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 14:15:04.920219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 14:15:04.920219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 14:15:04.920219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 14:15:04.920219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 14:15:04.920219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 14:15:04.920219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 14:15:04.920219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application conn.query(q) Oct 04 14:15:04.920219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 14:15:04.920219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 14:15:04.920219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 14:15:04.920219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application result.read() Oct 04 14:15:04.920219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return ref.to_dict() Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 14:15:04.930182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application next(self.gen) Oct 04 14:15:04.933537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 04 14:15:04.933537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application yield resource Oct 04 14:15:04.933537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 14:15:04.933537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application next(self.gen) Oct 04 14:15:04.933537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 04 14:15:04.933537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self.session.rollback() Oct 04 14:15:04.933537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 04 14:15:04.933537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self.force_reraise() Oct 04 14:15:04.933537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 04 14:15:04.933537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise self.value Oct 04 14:15:04.933537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 04 14:15:04.933537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 04 14:15:04.933537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 04 14:15:04.933537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application session.commit() Oct 04 14:15:04.933537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 04 14:15:04.933537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 04 14:15:04.933537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 04 14:15:04.938543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._prepare_impl() Oct 04 14:15:04.938543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 04 14:15:04.938543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self.session.flush() Oct 04 14:15:04.938543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 04 14:15:04.938543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._flush(objects) Oct 04 14:15:04.938543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 04 14:15:04.938543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 04 14:15:04.938543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 04 14:15:04.938543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application compat.raise_( Oct 04 14:15:04.938543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 14:15:04.938543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception Oct 04 14:15:04.938543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 04 14:15:04.938543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application flush_context.execute() Oct 04 14:15:04.938543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 04 14:15:04.938543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rec.execute(self) Oct 04 14:15:04.938543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 04 14:15:04.938543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 04 14:15:04.946605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 04 14:15:04.946605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 04 14:15:04.946605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 04 14:15:04.946605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 04 14:15:04.946605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 04 14:15:04.946605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 04 14:15:04.946605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 04 14:15:04.946605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 04 14:15:04.946605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 04 14:15:04.946605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 04 14:15:04.946605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 04 14:15:04.946605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 04 14:15:04.946605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 04 14:15:04.946605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 04 14:15:04.946605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 14:15:04.946605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception Oct 04 14:15:04.952500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 14:15:04.952500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 14:15:04.952500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 14:15:04.952500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 14:15:04.952500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 14:15:04.952500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 14:15:04.952500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 14:15:04.952500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application conn.query(q) Oct 04 14:15:04.952500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 14:15:04.952500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 14:15:04.952500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 14:15:04.952500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application result.read() Oct 04 14:15:04.952500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 14:15:04.952500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 14:15:04.952500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 14:15:04.952500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 14:15:04.958205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 14:15:04.958205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 14:15:04.958205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 14:15:04.958205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 14:15:04.958205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 04 14:15:04.958205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 04 14:15:04.958205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [parameters: {'id': '77dd595a7f4742a6987a00ae4f913745', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 04 14:15:04.958205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 04 14:15:04.958205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:04.958205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:15:04.958205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:04.958205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:04.958205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:04.958205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:04.958205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:04.958205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:04.971407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:04.971407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:04.971407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:04.971407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:04.971407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:04.971407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:04.971407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 04 14:15:04.971407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 04 14:15:04.971407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:04.971407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:04.971407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 04 14:15:04.971407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 04 14:15:04.971407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 04 14:15:04.971407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 04 14:15:04.971407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 04 14:15:04.971407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:04.975444 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 238/477] 10.222.0.64 () {60 vars in 1131 bytes} [Tue Oct 4 14:15:04 2022] POST /identity/v3/roles => generated 144 bytes in 369 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:04.975444 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650e8810-728a-45e8-86c1-9a210741bdbb None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:05.015257 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:05.145968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7752b389-1fa4-47eb-a363-37433e8c3d0f None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:05.205515 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:05.229975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a5c81f-f7fa-4612-97cd-1f6166201bd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:05.231839 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a5c81f-f7fa-4612-97cd-1f6166201bd4 None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:05.232610 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a5c81f-f7fa-4612-97cd-1f6166201bd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:05.233322 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a5c81f-f7fa-4612-97cd-1f6166201bd4 None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:05.234885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a5c81f-f7fa-4612-97cd-1f6166201bd4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:05.240613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a5c81f-f7fa-4612-97cd-1f6166201bd4 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:05.299851 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 240/478] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:15:04 2022] POST /identity/v3/users => generated 249 bytes in 438 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:15:05.331326 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650e8810-728a-45e8-86c1-9a210741bdbb None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:05.413400 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbd058a-eb94-4a4b-8805-794888110d79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:05.415057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbd058a-eb94-4a4b-8805-794888110d79 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:05.415823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbd058a-eb94-4a4b-8805-794888110d79 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:05.416552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbd058a-eb94-4a4b-8805-794888110d79 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:05.438884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbd058a-eb94-4a4b-8805-794888110d79 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:05.451682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-0bbd058a-eb94-4a4b-8805-794888110d79 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:15:05.451682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:05.451682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:05.451682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:05.451682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:05.451682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:05.451682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:05.451682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:05.451682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:05.451682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:05.451682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:05.451682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:05.451682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:05.451682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:05.451682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:05.451682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:05.453541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:05.453541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:05.453541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:05.453541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:05.453541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:05.453541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:05.453541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:05.453541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:05.453541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:05.453541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:05.453541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:05.453541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:05.453541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:05.453541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:05.453541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:05.453541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:05.456133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:05.456133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:05.456133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:05.456133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:05.456133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:05.456133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:05.456133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:05.456133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:05.456133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:05.456133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:05.456133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 14:15:05.456133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:05.461084 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 239/479] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:15:04 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 524 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:05.468007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a5c81f-f7fa-4612-97cd-1f6166201bd4 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:05.697410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a5c81f-f7fa-4612-97cd-1f6166201bd4 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:05.763731 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8883a51f-7ed1-4268-800c-cc5e910910ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:05.764731 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8883a51f-7ed1-4268-800c-cc5e910910ac None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:05.764731 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8883a51f-7ed1-4268-800c-cc5e910910ac None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:05.765033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8883a51f-7ed1-4268-800c-cc5e910910ac None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:05.765603 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8883a51f-7ed1-4268-800c-cc5e910910ac None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:05.765906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8883a51f-7ed1-4268-800c-cc5e910910ac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:05.769232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8883a51f-7ed1-4268-800c-cc5e910910ac None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:05.786080 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 241/480] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:05 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:06.716380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f7c7aa-ac26-44d2-a4c9-47da4627927d None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:06.718109 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f7c7aa-ac26-44d2-a4c9-47da4627927d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:06.718872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f7c7aa-ac26-44d2-a4c9-47da4627927d None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:06.723446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 240/481] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:06 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:06.729993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c40b3f-8a1e-484b-955a-3ad11c7bed46 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:06.730883 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c40b3f-8a1e-484b-955a-3ad11c7bed46 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:06.731593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c40b3f-8a1e-484b-955a-3ad11c7bed46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:06.773128 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-45c40b3f-8a1e-484b-955a-3ad11c7bed46 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:06.776689 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-45c40b3f-8a1e-484b-955a-3ad11c7bed46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:06.851752 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:06.938218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 242/482] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:07.317522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f7c7aa-ac26-44d2-a4c9-47da4627927d None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:07.441997 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:07.480736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f7c7aa-ac26-44d2-a4c9-47da4627927d None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:07.541331 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bd3faf-f709-4ce6-ae32-0fb2a651c893 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:07.542013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bd3faf-f709-4ce6-ae32-0fb2a651c893 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:07.542297 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bd3faf-f709-4ce6-ae32-0fb2a651c893 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:07.542579 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bd3faf-f709-4ce6-ae32-0fb2a651c893 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:07.561534 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41bd3faf-f709-4ce6-ae32-0fb2a651c893 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:07.567923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-41bd3faf-f709-4ce6-ae32-0fb2a651c893 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:07.567923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:07.567923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:07.567923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:07.567923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:07.567923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:07.567923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:07.567923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:07.567923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:07.567923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:07.567923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:07.567923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:07.567923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:07.567923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:07.567923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:07.567923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:07.568958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:07.568958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:07.568958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:07.568958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:07.568958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:07.568958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:07.568958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:07.568958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:07.568958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:07.568958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:07.568958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:07.568958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:07.568958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:07.568958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:07.568958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:07.568958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:07.569951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:07.569951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:07.569951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:07.569951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:07.569951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:07.569951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:07.569951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:07.569951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:07.569951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:07.569951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:07.569951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:07.569951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:07.569951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 241/483] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:07.576709 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c40b3f-8a1e-484b-955a-3ad11c7bed46 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:07.797137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c40b3f-8a1e-484b-955a-3ad11c7bed46 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:07.856035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5f31dd-23e8-4417-a656-1df042203c02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:07.857148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5f31dd-23e8-4417-a656-1df042203c02 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:07.857610 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5f31dd-23e8-4417-a656-1df042203c02 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:07.858035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5f31dd-23e8-4417-a656-1df042203c02 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:07.859399 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5f31dd-23e8-4417-a656-1df042203c02 None admin] RBAC: Adding query filter params (name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:07.860087 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5f31dd-23e8-4417-a656-1df042203c02 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:07.862920 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5f31dd-23e8-4417-a656-1df042203c02 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:07.879061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 243/484] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:15:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:07.881659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bd3faf-f709-4ce6-ae32-0fb2a651c893 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:08.033509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e85360-bc0e-4c2f-a3b7-25fd1d89323a None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:08.034031 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e85360-bc0e-4c2f-a3b7-25fd1d89323a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:08.034500 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e85360-bc0e-4c2f-a3b7-25fd1d89323a None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:08.035916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 244/485] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:15:08.050293 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04750ad8-ef98-4aec-a561-5a07106c4ecc None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:08.050585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04750ad8-ef98-4aec-a561-5a07106c4ecc None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:08.050827 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04750ad8-ef98-4aec-a561-5a07106c4ecc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:08.085705 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-04750ad8-ef98-4aec-a561-5a07106c4ecc None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:08.086976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-04750ad8-ef98-4aec-a561-5a07106c4ecc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:08.122300 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:08.156219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bd3faf-f709-4ce6-ae32-0fb2a651c893 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:08.240388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 245/486] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:08.279024 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4d85d5-84f0-4a23-835a-bc3a14332a1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:08.280936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4d85d5-84f0-4a23-835a-bc3a14332a1b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:08.281685 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4d85d5-84f0-4a23-835a-bc3a14332a1b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:08.282431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4d85d5-84f0-4a23-835a-bc3a14332a1b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:08.300852 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:08.305201 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4d85d5-84f0-4a23-835a-bc3a14332a1b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:08.308851 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4d85d5-84f0-4a23-835a-bc3a14332a1b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:08.343210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-ca4d85d5-84f0-4a23-835a-bc3a14332a1b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:08.343210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:08.343210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:08.343210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:08.343210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:08.343210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:08.343210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:08.343210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:08.343210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:08.343210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:08.343210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:08.343210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:08.343210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:08.343210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:08.343210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:08.343210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:08.344222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:08.344222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:08.344222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:08.344222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:08.344222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:08.344222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:08.344222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:08.344222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:08.344222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:08.344222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:08.344222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:08.344222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:08.344222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:08.344222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:08.344222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:08.344222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:08.344222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:08.345372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:08.345372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:08.345372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:08.349812 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 242/487] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 469 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:08.359768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04750ad8-ef98-4aec-a561-5a07106c4ecc None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:08.609494 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04750ad8-ef98-4aec-a561-5a07106c4ecc None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:08.640360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4d85d5-84f0-4a23-835a-bc3a14332a1b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:08.674974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629b21ca-1c01-42a0-aa03-e7e71724af3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:08.675989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629b21ca-1c01-42a0-aa03-e7e71724af3c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:08.676506 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629b21ca-1c01-42a0-aa03-e7e71724af3c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:08.676976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629b21ca-1c01-42a0-aa03-e7e71724af3c None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:08.677876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629b21ca-1c01-42a0-aa03-e7e71724af3c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:08.678383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629b21ca-1c01-42a0-aa03-e7e71724af3c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:08.680017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629b21ca-1c01-42a0-aa03-e7e71724af3c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:08.700466 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 246/488] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 341 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:08.702383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd418f2-02d1-4c24-a543-ba06fed7c9c6 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:08.703056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd418f2-02d1-4c24-a543-ba06fed7c9c6 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:08.703563 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd418f2-02d1-4c24-a543-ba06fed7c9c6 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:08.708015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 247/489] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:08 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:08.709322 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd418f2-02d1-4c24-a543-ba06fed7c9c6 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:08.779110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:08.841743 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4d85d5-84f0-4a23-835a-bc3a14332a1b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:08.896776 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd418f2-02d1-4c24-a543-ba06fed7c9c6 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:08.922994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f372c8-1ec8-43e4-a67c-59f3c23728f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:08.924400 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f372c8-1ec8-43e4-a67c-59f3c23728f3 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:08.924861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f372c8-1ec8-43e4-a67c-59f3c23728f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:08.925275 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f372c8-1ec8-43e4-a67c-59f3c23728f3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:08.939847 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f372c8-1ec8-43e4-a67c-59f3c23728f3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:08.948669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-01f372c8-1ec8-43e4-a67c-59f3c23728f3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:08.948669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:08.948669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:08.948669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:08.948669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:08.948669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:08.948669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:08.948669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:08.948669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:08.948669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:08.948669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:08.948669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:08.948669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:08.948669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:08.948669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:08.948669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:08.949735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:08.949735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:08.949735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:08.949735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:08.949735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:08.949735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:08.949735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:08.949735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:08.949735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:08.949735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:08.949735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:08.949735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:08.949735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:08.949735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:08.949735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:08.949735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:08.950555 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:08.950555 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:08.950555 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:08.950555 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:08.950555 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:08.950555 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:08.950555 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:08.950555 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:08.950555 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:08.950555 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:08.950555 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:08.950555 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:08.958814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 243/490] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:08.961110 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44637a6a-3531-49ef-9aaa-9d78530f2b5f None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:08.961637 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44637a6a-3531-49ef-9aaa-9d78530f2b5f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:08.962292 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44637a6a-3531-49ef-9aaa-9d78530f2b5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:08.969518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30cd4bf-af2f-4e4b-b59c-e253ab5cd747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:08.971793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30cd4bf-af2f-4e4b-b59c-e253ab5cd747 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:08.972432 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30cd4bf-af2f-4e4b-b59c-e253ab5cd747 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:08.974417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30cd4bf-af2f-4e4b-b59c-e253ab5cd747 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:08.987144 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-44637a6a-3531-49ef-9aaa-9d78530f2b5f None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:08.988949 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-44637a6a-3531-49ef-9aaa-9d78530f2b5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:08.997119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30cd4bf-af2f-4e4b-b59c-e253ab5cd747 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:09.000296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30cd4bf-af2f-4e4b-b59c-e253ab5cd747 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:09.028605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-f30cd4bf-af2f-4e4b-b59c-e253ab5cd747 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 04 14:15:09.028605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:09.028605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:09.028605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:09.028605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:09.028605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:09.028605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:09.028605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:09.028605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:09.028605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:09.028605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:09.028605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:09.028605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:09.028605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:09.028605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:09.028605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:09.029641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:09.030866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:09.030866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:09.030866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:09.030866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:09.030866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:09.030866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:09.030866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:09.030866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:09.030866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 04 14:15:09.030866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:09.036883 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 248/491] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:15:08 2022] GET /identity/v3/users/glance => generated 84 bytes in 322 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:09.038454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30cd4bf-af2f-4e4b-b59c-e253ab5cd747 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:09.059154 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:09.097266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 244/492] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:09.102888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44637a6a-3531-49ef-9aaa-9d78530f2b5f None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:09.283206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:09.321588 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30cd4bf-af2f-4e4b-b59c-e253ab5cd747 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:09.396343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44637a6a-3531-49ef-9aaa-9d78530f2b5f None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:09.396550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6216458e-f74a-4471-9098-9dafd49547ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:09.396550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6216458e-f74a-4471-9098-9dafd49547ff None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:09.396807 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6216458e-f74a-4471-9098-9dafd49547ff None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:09.397255 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6216458e-f74a-4471-9098-9dafd49547ff None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:09.398435 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6216458e-f74a-4471-9098-9dafd49547ff None admin] RBAC: Adding query filter params (name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:09.398918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6216458e-f74a-4471-9098-9dafd49547ff None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:09.401457 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6216458e-f74a-4471-9098-9dafd49547ff None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:09.420229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 249/493] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:15:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 383 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:09.422431 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01975dd-755d-40cd-bee5-a6b84df192a5 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:09.423683 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01975dd-755d-40cd-bee5-a6b84df192a5 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:09.424148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01975dd-755d-40cd-bee5-a6b84df192a5 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:09.426908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 250/494] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:09 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:09.428212 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01975dd-755d-40cd-bee5-a6b84df192a5 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:09.469046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e60357-06dc-4823-ad93-759441f911fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:09.470449 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e60357-06dc-4823-ad93-759441f911fb None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:09.470884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e60357-06dc-4823-ad93-759441f911fb None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:09.471321 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e60357-06dc-4823-ad93-759441f911fb None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:09.472334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e60357-06dc-4823-ad93-759441f911fb None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:09.472774 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e60357-06dc-4823-ad93-759441f911fb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:09.474612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e60357-06dc-4823-ad93-759441f911fb None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:09.530720 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 245/495] 10.222.0.64 () {56 vars in 1137 bytes} [Tue Oct 4 14:15:09 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 427 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:09.530720 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174e7104-ab31-4171-ae2b-7f786f8dd4c2 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:09.530720 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174e7104-ab31-4171-ae2b-7f786f8dd4c2 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:09.530720 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174e7104-ab31-4171-ae2b-7f786f8dd4c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:09.557951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-174e7104-ab31-4171-ae2b-7f786f8dd4c2 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:09.559618 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-174e7104-ab31-4171-ae2b-7f786f8dd4c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:09.605727 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01975dd-755d-40cd-bee5-a6b84df192a5 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:09.616427 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:09.653411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb300659-e678-4091-b659-e61d3b0c0e3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:09.654047 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb300659-e678-4091-b659-e61d3b0c0e3c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:09.654331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb300659-e678-4091-b659-e61d3b0c0e3c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:09.654597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb300659-e678-4091-b659-e61d3b0c0e3c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:09.670839 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb300659-e678-4091-b659-e61d3b0c0e3c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:09.672809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb300659-e678-4091-b659-e61d3b0c0e3c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:09.673952 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 246/496] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:09.674920 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174e7104-ab31-4171-ae2b-7f786f8dd4c2 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:09.688638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:09.716620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-fb300659-e678-4091-b659-e61d3b0c0e3c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:09.716620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:09.716620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:09.716620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:09.716620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:09.716620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:09.716620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:09.716620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:09.716620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:09.716620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:09.716620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:09.716620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:09.716620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:09.716620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:09.716620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:09.716620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:09.718342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:09.718342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:09.718342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:09.718342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:09.718342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:09.718342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:09.718342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:09.718342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:09.718342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:09.718342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:09.718342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:09.718342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:09.718342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:09.718342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:09.718342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:09.718342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:09.718342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:09.720550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:09.720550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:09.720550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:09.727140 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 251/497] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:09.728651 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb300659-e678-4091-b659-e61d3b0c0e3c None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:09.888152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174e7104-ab31-4171-ae2b-7f786f8dd4c2 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:09.918019 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb300659-e678-4091-b659-e61d3b0c0e3c None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:09.974098 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098f67c0-3329-41e8-a287-7c155c71aaac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:09.975115 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098f67c0-3329-41e8-a287-7c155c71aaac None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:09.975561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098f67c0-3329-41e8-a287-7c155c71aaac None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:09.976005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098f67c0-3329-41e8-a287-7c155c71aaac None admin] PATH_INFO: `/v3/roles/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:09.987040 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9484c69-23fc-4181-a201-a6f3daba887b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:09.988300 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9484c69-23fc-4181-a201-a6f3daba887b None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:09.988983 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9484c69-23fc-4181-a201-a6f3daba887b None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:09.989614 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9484c69-23fc-4181-a201-a6f3daba887b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:09.995149 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098f67c0-3329-41e8-a287-7c155c71aaac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:10.004521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-098f67c0-3329-41e8-a287-7c155c71aaac None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:10.004521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:10.004521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:10.004521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:10.004521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:10.004521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:10.004521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:10.004521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:10.004521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:10.004521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:10.004521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:10.004521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:10.004521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:10.004521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:10.004521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:10.004521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:10.005509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:10.005509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:10.005509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:10.005509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:10.005509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:10.005509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:10.005509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:10.005509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:10.005509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:10.005509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:10.005509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:10.005509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:10.005509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:10.005509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:10.005509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:10.005509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:10.006309 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:10.006309 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:10.006309 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:10.006309 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:10.006309 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:10.006309 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:10.006309 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:10.006309 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:10.006309 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:10.006309 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:10.006309 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:10.006309 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:10.006937 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 247/498] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 332 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:10.010038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098f67c0-3329-41e8-a287-7c155c71aaac None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:10.059858 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:10.097938 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:10.107997 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9484c69-23fc-4181-a201-a6f3daba887b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:10.111013 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9484c69-23fc-4181-a201-a6f3daba887b None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:10.176854 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098f67c0-3329-41e8-a287-7c155c71aaac None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:10.228897 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9011bfaf-54d5-46b0-8776-65dede16719d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:10.229847 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9011bfaf-54d5-46b0-8776-65dede16719d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:10.230206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9011bfaf-54d5-46b0-8776-65dede16719d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:10.230609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9011bfaf-54d5-46b0-8776-65dede16719d None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:10.231675 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9011bfaf-54d5-46b0-8776-65dede16719d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:10.232114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9011bfaf-54d5-46b0-8776-65dede16719d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:10.234167 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9011bfaf-54d5-46b0-8776-65dede16719d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:10.247947 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 252/499] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:15:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 514 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:15:10.247947 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9484c69-23fc-4181-a201-a6f3daba887b None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:10.258140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 248/500] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:10.264413 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9011bfaf-54d5-46b0-8776-65dede16719d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:10.383403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:10.396424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9484c69-23fc-4181-a201-a6f3daba887b None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:10.410138 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:10.425959 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9011bfaf-54d5-46b0-8776-65dede16719d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:10.460262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aec3820-f79e-4750-b6e1-b9919c499db4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:10.461413 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aec3820-f79e-4750-b6e1-b9919c499db4 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:10.461888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aec3820-f79e-4750-b6e1-b9919c499db4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:10.462355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aec3820-f79e-4750-b6e1-b9919c499db4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:10.466063 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33f6f27-24f6-4be4-9b6e-1f806e89eee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:10.466683 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33f6f27-24f6-4be4-9b6e-1f806e89eee9 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:10.466966 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33f6f27-24f6-4be4-9b6e-1f806e89eee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:10.467238 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33f6f27-24f6-4be4-9b6e-1f806e89eee9 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:10.468990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33f6f27-24f6-4be4-9b6e-1f806e89eee9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:10.469303 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33f6f27-24f6-4be4-9b6e-1f806e89eee9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:10.472753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33f6f27-24f6-4be4-9b6e-1f806e89eee9 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:10.482166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 249/501] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:15:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:10.483049 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33f6f27-24f6-4be4-9b6e-1f806e89eee9 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:10.485480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aec3820-f79e-4750-b6e1-b9919c499db4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:10.496417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-5aec3820-f79e-4750-b6e1-b9919c499db4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:10.496417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:10.496417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:10.496417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:10.496417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:10.496417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:10.496417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:10.496417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:10.496417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:10.496417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:10.496417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:10.496417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:10.496417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:10.496417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:10.496417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:10.496417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:10.498199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:10.498199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:10.498199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:10.498199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:10.498199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:10.498199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:10.498199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:10.498199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:10.498199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:10.498199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:10.498199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:10.498199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:10.498199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:10.498199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:10.498199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:10.498199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:10.499194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:10.499194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:10.499194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:10.499194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:10.499194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:10.499194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:10.499194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:10.499194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:10.499194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:10.499194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:10.499194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:10.499194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:10.499194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 253/502] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:10.499866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aec3820-f79e-4750-b6e1-b9919c499db4 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:10.624877 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33f6f27-24f6-4be4-9b6e-1f806e89eee9 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:10.663298 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aec3820-f79e-4750-b6e1-b9919c499db4 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:10.676147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e18070c-9172-40ac-bb12-5aa97e88b9ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:10.677177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e18070c-9172-40ac-bb12-5aa97e88b9ff None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:10.677608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e18070c-9172-40ac-bb12-5aa97e88b9ff None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:10.678036 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e18070c-9172-40ac-bb12-5aa97e88b9ff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:10.696925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e18070c-9172-40ac-bb12-5aa97e88b9ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:10.699177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e18070c-9172-40ac-bb12-5aa97e88b9ff None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:10.723192 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80b1616-62f9-4c44-bfb0-8c68fdaad52e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:10.724034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80b1616-62f9-4c44-bfb0-8c68fdaad52e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:10.724376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-5e18070c-9172-40ac-bb12-5aa97e88b9ff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:10.724376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:10.724376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:10.724376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:10.724376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:10.724376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:10.724376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:10.724376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:10.724376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:10.724376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:10.724376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:10.724376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:10.724376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:10.724376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:10.724376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:10.724376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:10.726362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:10.726362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:10.726362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:10.726362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:10.726362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:10.726362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:10.726362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:10.726362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:10.726362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:10.726362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:10.726362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:10.726362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:10.726362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:10.726362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:10.726362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:10.726362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:10.726362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:10.728027 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:10.728027 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:10.728027 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:10.728386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80b1616-62f9-4c44-bfb0-8c68fdaad52e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:10.728386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80b1616-62f9-4c44-bfb0-8c68fdaad52e None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:10.728598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 250/503] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:10.728598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f76fd5-fde5-4bd5-a81d-26b92242d683 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:10.728868 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f76fd5-fde5-4bd5-a81d-26b92242d683 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:10.729254 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f76fd5-fde5-4bd5-a81d-26b92242d683 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:10.731964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 251/504] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:10.733630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f76fd5-fde5-4bd5-a81d-26b92242d683 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:10.739519 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:10.743637 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80b1616-62f9-4c44-bfb0-8c68fdaad52e None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:10.746324 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80b1616-62f9-4c44-bfb0-8c68fdaad52e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:10.769583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-b80b1616-62f9-4c44-bfb0-8c68fdaad52e None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 14:15:10.769583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:10.769583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:10.769583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:10.769583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:10.769583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:10.769583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:10.769583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:10.769583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:10.769583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:10.769583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:10.769583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:10.769583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:10.769583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:10.769583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:10.769583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:10.770767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:10.772616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:10.772616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:10.772616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:10.772616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:10.772616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:10.772616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:10.772616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:10.772616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:10.772616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 14:15:10.772616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:10.772616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 254/505] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:15:10 2022] GET /identity/v3/users/cinder => generated 84 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:10.773740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80b1616-62f9-4c44-bfb0-8c68fdaad52e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:10.807144 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:10.884557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f76fd5-fde5-4bd5-a81d-26b92242d683 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:10.929417 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09f30dc-698d-4dd0-b677-cfc4925af3b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:10.930424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09f30dc-698d-4dd0-b677-cfc4925af3b3 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:10.930777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09f30dc-698d-4dd0-b677-cfc4925af3b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:10.931098 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09f30dc-698d-4dd0-b677-cfc4925af3b3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:10.951041 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09f30dc-698d-4dd0-b677-cfc4925af3b3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:10.951310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09f30dc-698d-4dd0-b677-cfc4925af3b3 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:10.951827 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80b1616-62f9-4c44-bfb0-8c68fdaad52e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:10.974606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-c09f30dc-698d-4dd0-b677-cfc4925af3b3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:10.974606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:10.974606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:10.974606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:10.974606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:10.974606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:10.974606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:10.974606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:10.974606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:10.974606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:10.974606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:10.974606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:10.974606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:10.974606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:10.974606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:10.974606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:10.977060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:10.977060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:10.977060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:10.977060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:10.977060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:10.977060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:10.977060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:10.977060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:10.977060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:10.977060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:10.977060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:10.977060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:10.977060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:10.977060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:10.977060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:10.977060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:10.977060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:10.978727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:10.978727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:10.978727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:10.978727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 252/506] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:10.978727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09f30dc-698d-4dd0-b677-cfc4925af3b3 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:11.012089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a80d95-20ce-4ff9-8bef-432b2af99fb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:11.012089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a80d95-20ce-4ff9-8bef-432b2af99fb2 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:11.012089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a80d95-20ce-4ff9-8bef-432b2af99fb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:11.012089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a80d95-20ce-4ff9-8bef-432b2af99fb2 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:11.012845 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a80d95-20ce-4ff9-8bef-432b2af99fb2 None admin] RBAC: Adding query filter params (name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:11.012845 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a80d95-20ce-4ff9-8bef-432b2af99fb2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:11.014127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a80d95-20ce-4ff9-8bef-432b2af99fb2 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:11.029393 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 255/507] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:15:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:11.031358 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab4dfc6-13e8-4586-960d-df4969b68193 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:11.031899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab4dfc6-13e8-4586-960d-df4969b68193 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:11.033199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab4dfc6-13e8-4586-960d-df4969b68193 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:11.060828 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-3ab4dfc6-13e8-4586-960d-df4969b68193 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:11.062272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-3ab4dfc6-13e8-4586-960d-df4969b68193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:11.071258 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:11.140887 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09f30dc-698d-4dd0-b677-cfc4925af3b3 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:11.169203 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 256/508] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:11.170610 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab4dfc6-13e8-4586-960d-df4969b68193 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:11.196413 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:11.199577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a58e0d-24cb-4575-a5e1-6caf9d106afd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:11.200405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a58e0d-24cb-4575-a5e1-6caf9d106afd None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:11.200810 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a58e0d-24cb-4575-a5e1-6caf9d106afd None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:11.200997 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a58e0d-24cb-4575-a5e1-6caf9d106afd None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:11.201813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a58e0d-24cb-4575-a5e1-6caf9d106afd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:11.202126 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a58e0d-24cb-4575-a5e1-6caf9d106afd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:11.203506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a58e0d-24cb-4575-a5e1-6caf9d106afd None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:11.228499 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 253/509] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:11.228499 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a58e0d-24cb-4575-a5e1-6caf9d106afd None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:11.361919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab4dfc6-13e8-4586-960d-df4969b68193 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:11.412140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a58e0d-24cb-4575-a5e1-6caf9d106afd None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:11.429615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1b7d4e-57fd-4476-82b4-b75b57911ecc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:11.430836 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1b7d4e-57fd-4476-82b4-b75b57911ecc None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:11.431214 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1b7d4e-57fd-4476-82b4-b75b57911ecc None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:11.431773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1b7d4e-57fd-4476-82b4-b75b57911ecc None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:11.433482 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1b7d4e-57fd-4476-82b4-b75b57911ecc None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:11.434271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1b7d4e-57fd-4476-82b4-b75b57911ecc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:11.436892 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:11.437087 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1b7d4e-57fd-4476-82b4-b75b57911ecc None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:11.472944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20567aec-cec6-425a-b631-b7cf3cadf3c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:11.473581 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20567aec-cec6-425a-b631-b7cf3cadf3c2 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:11.473862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20567aec-cec6-425a-b631-b7cf3cadf3c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:11.474135 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20567aec-cec6-425a-b631-b7cf3cadf3c2 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:11.474980 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20567aec-cec6-425a-b631-b7cf3cadf3c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:11.475283 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20567aec-cec6-425a-b631-b7cf3cadf3c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:11.477011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20567aec-cec6-425a-b631-b7cf3cadf3c2 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:11.499438 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 254/510] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:11.499438 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20567aec-cec6-425a-b631-b7cf3cadf3c2 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:11.508707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 257/511] 10.222.0.64 () {56 vars in 1137 bytes} [Tue Oct 4 14:15:11 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 339 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:11.510769 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1b7d4e-57fd-4476-82b4-b75b57911ecc None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:11.636394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:11.661919 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20567aec-cec6-425a-b631-b7cf3cadf3c2 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:11.699894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1b7d4e-57fd-4476-82b4-b75b57911ecc None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:11.734106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da79dd25-ec5d-448a-97d9-5b4a2df978c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:11.734889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da79dd25-ec5d-448a-97d9-5b4a2df978c8 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:11.735850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da79dd25-ec5d-448a-97d9-5b4a2df978c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:11.736352 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da79dd25-ec5d-448a-97d9-5b4a2df978c8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:11.748681 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a279eb8d-90b9-480b-b5d1-3b6d7a24d600 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:11.749344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a279eb8d-90b9-480b-b5d1-3b6d7a24d600 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:11.749703 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a279eb8d-90b9-480b-b5d1-3b6d7a24d600 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:11.749911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a279eb8d-90b9-480b-b5d1-3b6d7a24d600 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:11.752141 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da79dd25-ec5d-448a-97d9-5b4a2df978c8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:11.754073 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da79dd25-ec5d-448a-97d9-5b4a2df978c8 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:11.758222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a279eb8d-90b9-480b-b5d1-3b6d7a24d600 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:11.759745 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a279eb8d-90b9-480b-b5d1-3b6d7a24d600 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:11.774229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-da79dd25-ec5d-448a-97d9-5b4a2df978c8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:11.774229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:11.774229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:11.774229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:11.774229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:11.774229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:11.774229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:11.774229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:11.774229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:11.774229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:11.774229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:11.774229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:11.774229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:11.774229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:11.774229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:11.774229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:11.776181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:11.776181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:11.776181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:11.776181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:11.776181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:11.776181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:11.776181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:11.776181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:11.776181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:11.776181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:11.776181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:11.776181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:11.776181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:11.776181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:11.776181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:11.776181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:11.776181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:11.778017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:11.778017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:11.778017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:11.778017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 255/512] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:11.778017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da79dd25-ec5d-448a-97d9-5b4a2df978c8 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:11.778644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-a279eb8d-90b9-480b-b5d1-3b6d7a24d600 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:11.778644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:11.778644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:11.778644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:11.778644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:11.778644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:11.778644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:11.778644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:11.778644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:11.778644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:11.778644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:11.778644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:11.778644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:11.778644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:11.778644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:11.781736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:11.781736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:11.781736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:11.781736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:11.781736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:11.781736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:11.781736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:11.781736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:11.781736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:11.781736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:11.781736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:11.781736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:11.781736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:11.781736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:11.781736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:11.781736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:11.781736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:11.786902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:11.786902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:11.786902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:11.786902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:11.786902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:11.786902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:11.786902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:11.786902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:11.786902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 258/513] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:11.786902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a279eb8d-90b9-480b-b5d1-3b6d7a24d600 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:11.824873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:11.917211 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a279eb8d-90b9-480b-b5d1-3b6d7a24d600 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:11.955509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da79dd25-ec5d-448a-97d9-5b4a2df978c8 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:11.957228 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:11.979587 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b779f807-3924-4d2d-8ee2-8202d468d8f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:11.980369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b779f807-3924-4d2d-8ee2-8202d468d8f0 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:11.980827 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b779f807-3924-4d2d-8ee2-8202d468d8f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:11.981208 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b779f807-3924-4d2d-8ee2-8202d468d8f0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:12.003949 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282cab5c-20fe-4012-ba81-7a123f2601ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:12.004854 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282cab5c-20fe-4012-ba81-7a123f2601ed None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:12.005290 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282cab5c-20fe-4012-ba81-7a123f2601ed None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:12.005731 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282cab5c-20fe-4012-ba81-7a123f2601ed None admin] PATH_INFO: `/v3/users/nova` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:12.018705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282cab5c-20fe-4012-ba81-7a123f2601ed None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:12.021298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282cab5c-20fe-4012-ba81-7a123f2601ed None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:12.041513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-282cab5c-20fe-4012-ba81-7a123f2601ed None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:15:12.041513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:12.041513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:12.041513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:12.041513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:12.041513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:12.041513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:12.041513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:12.041513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:12.041513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:12.041513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:12.041513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:12.041513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:12.041513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:12.041513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:12.041513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:12.043405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:12.045175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:12.045175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:12.045175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:12.045175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:12.045175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:12.045175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:12.045175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:12.045175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:12.045175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:15:12.045175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:12.045175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 256/514] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Oct 4 14:15:11 2022] GET /identity/v3/users/nova => generated 82 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:12.046441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282cab5c-20fe-4012-ba81-7a123f2601ed None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:12.082277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b779f807-3924-4d2d-8ee2-8202d468d8f0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:12.083790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b779f807-3924-4d2d-8ee2-8202d468d8f0 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:12.211246 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:12.214451 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 259/515] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:15:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:15:12.215463 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b779f807-3924-4d2d-8ee2-8202d468d8f0 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:12.238807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282cab5c-20fe-4012-ba81-7a123f2601ed None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:12.254159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:12.296867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c03ce1-0a6c-434b-85a5-21bd96c0bbbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:12.298030 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c03ce1-0a6c-434b-85a5-21bd96c0bbbe None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:12.298687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c03ce1-0a6c-434b-85a5-21bd96c0bbbe None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:12.299172 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c03ce1-0a6c-434b-85a5-21bd96c0bbbe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:12.320091 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c03ce1-0a6c-434b-85a5-21bd96c0bbbe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:12.323010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c03ce1-0a6c-434b-85a5-21bd96c0bbbe None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:12.349223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-c3c03ce1-0a6c-434b-85a5-21bd96c0bbbe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:12.349223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:12.349223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:12.349223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:12.349223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:12.349223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:12.349223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:12.349223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:12.349223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:12.349223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:12.349223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:12.349223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:12.349223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:12.349223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:12.349223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:12.349223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:12.350271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:12.350271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:12.350271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:12.350271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:12.350271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:12.350271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:12.350271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:12.350271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:12.350271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:12.350271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:12.350271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:12.350271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:12.350271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:12.350271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:12.350271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:12.350271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:12.350271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:12.351124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:12.351124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:12.351124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:12.351287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 257/516] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 307 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:12.352792 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c03ce1-0a6c-434b-85a5-21bd96c0bbbe None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:12.358831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b779f807-3924-4d2d-8ee2-8202d468d8f0 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:12.407843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccffdb51-fe11-4d84-bd40-a29be58528c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:12.409079 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccffdb51-fe11-4d84-bd40-a29be58528c4 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:12.409553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccffdb51-fe11-4d84-bd40-a29be58528c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:12.410040 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccffdb51-fe11-4d84-bd40-a29be58528c4 None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:12.411354 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccffdb51-fe11-4d84-bd40-a29be58528c4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:12.411911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccffdb51-fe11-4d84-bd40-a29be58528c4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:12.414823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccffdb51-fe11-4d84-bd40-a29be58528c4 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:12.447522 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 260/517] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:12.449097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccffdb51-fe11-4d84-bd40-a29be58528c4 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:12.491322 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c03ce1-0a6c-434b-85a5-21bd96c0bbbe None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:12.531727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83738f3-33bc-46b7-84bf-0d78c9a4f9fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:12.533256 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83738f3-33bc-46b7-84bf-0d78c9a4f9fd None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:12.533256 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83738f3-33bc-46b7-84bf-0d78c9a4f9fd None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:12.533539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83738f3-33bc-46b7-84bf-0d78c9a4f9fd None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:12.534055 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83738f3-33bc-46b7-84bf-0d78c9a4f9fd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:12.534360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83738f3-33bc-46b7-84bf-0d78c9a4f9fd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:12.535864 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83738f3-33bc-46b7-84bf-0d78c9a4f9fd None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:12.543757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:12.550537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 258/518] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:12.551410 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83738f3-33bc-46b7-84bf-0d78c9a4f9fd None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:12.600345 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:12.619539 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccffdb51-fe11-4d84-bd40-a29be58528c4 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:12.668454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d5a1d4-25d7-43c2-b870-0701eb58eba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:12.669080 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d5a1d4-25d7-43c2-b870-0701eb58eba4 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:12.669355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d5a1d4-25d7-43c2-b870-0701eb58eba4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:12.669629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d5a1d4-25d7-43c2-b870-0701eb58eba4 None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:12.670379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d5a1d4-25d7-43c2-b870-0701eb58eba4 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:12.670724 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d5a1d4-25d7-43c2-b870-0701eb58eba4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:12.673380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d5a1d4-25d7-43c2-b870-0701eb58eba4 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:12.689689 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83738f3-33bc-46b7-84bf-0d78c9a4f9fd None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:12.694872 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 261/519] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Oct 4 14:15:12 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:12.695568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d5a1d4-25d7-43c2-b870-0701eb58eba4 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:12.739708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6ce57a-c652-4203-b6f6-494238d4366d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:12.740578 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6ce57a-c652-4203-b6f6-494238d4366d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:12.740899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6ce57a-c652-4203-b6f6-494238d4366d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:12.741209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6ce57a-c652-4203-b6f6-494238d4366d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:12.758061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6ce57a-c652-4203-b6f6-494238d4366d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:12.759997 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6ce57a-c652-4203-b6f6-494238d4366d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:12.778592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-1d6ce57a-c652-4203-b6f6-494238d4366d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:12.778592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:12.778592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:12.778592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:12.778592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:12.778592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:12.778592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:12.778592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:12.778592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:12.778592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:12.778592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:12.778592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:12.778592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:12.778592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:12.778592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:12.778592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:12.780875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:12.780875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:12.780875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:12.780875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:12.780875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:12.780875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:12.780875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:12.780875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:12.780875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:12.780875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:12.780875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:12.780875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:12.780875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:12.780875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:12.780875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:12.780875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:12.780875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:12.783449 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:12.783449 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:12.783449 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:12.783449 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 259/520] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:12.783449 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6ce57a-c652-4203-b6f6-494238d4366d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:12.830457 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d5a1d4-25d7-43c2-b870-0701eb58eba4 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:12.865442 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:12.870644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b590e72c-121a-4342-b65d-2dedb95c59de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:12.871352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b590e72c-121a-4342-b65d-2dedb95c59de None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:12.871655 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b590e72c-121a-4342-b65d-2dedb95c59de None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:12.871940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b590e72c-121a-4342-b65d-2dedb95c59de None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:12.872632 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b590e72c-121a-4342-b65d-2dedb95c59de None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:12.872941 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b590e72c-121a-4342-b65d-2dedb95c59de None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:12.874377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b590e72c-121a-4342-b65d-2dedb95c59de None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:12.888717 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 262/521] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:12.889595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b590e72c-121a-4342-b65d-2dedb95c59de None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:12.894528 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:12.968419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6ce57a-c652-4203-b6f6-494238d4366d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:13.009857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b590e72c-121a-4342-b65d-2dedb95c59de None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:13.026838 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9a1159-a2eb-4e30-a836-f92e32dcc556 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:13.027771 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9a1159-a2eb-4e30-a836-f92e32dcc556 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:13.028101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9a1159-a2eb-4e30-a836-f92e32dcc556 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:13.028551 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9a1159-a2eb-4e30-a836-f92e32dcc556 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:13.029437 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9a1159-a2eb-4e30-a836-f92e32dcc556 None admin] RBAC: Adding query filter params (role.id=3ebe3c0be8c34c5eae533311e9b44176, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=35a5a4ef40204e158649ca25583bbde7) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:13.029812 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9a1159-a2eb-4e30-a836-f92e32dcc556 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:13.032488 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9a1159-a2eb-4e30-a836-f92e32dcc556 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:13.041023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 260/522] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:15:12 2022] GET /identity/v3/role_assignments?user.id=35a5a4ef40204e158649ca25583bbde7&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=3ebe3c0be8c34c5eae533311e9b44176 => generated 259 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:13.042719 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9a1159-a2eb-4e30-a836-f92e32dcc556 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:13.077127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504f0a37-3e59-4a0e-a5d6-c6018a15108e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:13.078239 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504f0a37-3e59-4a0e-a5d6-c6018a15108e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:13.079470 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504f0a37-3e59-4a0e-a5d6-c6018a15108e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:13.079743 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504f0a37-3e59-4a0e-a5d6-c6018a15108e None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:13.089756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504f0a37-3e59-4a0e-a5d6-c6018a15108e None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:13.091299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504f0a37-3e59-4a0e-a5d6-c6018a15108e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:13.106518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-504f0a37-3e59-4a0e-a5d6-c6018a15108e None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 14:15:13.106518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:13.106518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:13.106518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:13.106518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:13.106518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:13.106518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:13.106518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:13.106518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:13.106518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:13.106518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:13.106518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:13.106518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:13.106518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:13.106518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:13.106518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:13.108660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:13.111326 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:13.111326 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:13.111326 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:13.111326 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:13.111326 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:13.111326 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:13.111326 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:13.111326 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:13.111326 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 14:15:13.111326 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:13.113491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 263/523] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:12 2022] GET /identity/v3/users/neutron => generated 85 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:13.115290 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504f0a37-3e59-4a0e-a5d6-c6018a15108e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:13.212946 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:13.246845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9a1159-a2eb-4e30-a836-f92e32dcc556 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:13.269820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:13.294734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504f0a37-3e59-4a0e-a5d6-c6018a15108e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:13.311067 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fc5605-3809-4a37-9ea3-d68e31a61c1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:13.312085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fc5605-3809-4a37-9ea3-d68e31a61c1d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:13.312416 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fc5605-3809-4a37-9ea3-d68e31a61c1d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:13.312694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fc5605-3809-4a37-9ea3-d68e31a61c1d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:13.341483 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa123b9-972e-4215-9176-51cf65fb6574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:13.342111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa123b9-972e-4215-9176-51cf65fb6574 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:13.342391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa123b9-972e-4215-9176-51cf65fb6574 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:13.342695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa123b9-972e-4215-9176-51cf65fb6574 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:13.343363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa123b9-972e-4215-9176-51cf65fb6574 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:13.343671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa123b9-972e-4215-9176-51cf65fb6574 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:13.345775 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa123b9-972e-4215-9176-51cf65fb6574 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:13.367546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 264/524] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:13.367546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa123b9-972e-4215-9176-51cf65fb6574 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:13.418628 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fc5605-3809-4a37-9ea3-d68e31a61c1d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:13.420762 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fc5605-3809-4a37-9ea3-d68e31a61c1d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:13.526771 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa123b9-972e-4215-9176-51cf65fb6574 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:13.559155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:13.574848 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631da1cc-1ef2-4957-be4c-d91ca507fa9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:13.576014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631da1cc-1ef2-4957-be4c-d91ca507fa9e None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:13.576487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631da1cc-1ef2-4957-be4c-d91ca507fa9e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:13.576908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631da1cc-1ef2-4957-be4c-d91ca507fa9e None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:13.577725 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631da1cc-1ef2-4957-be4c-d91ca507fa9e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:13.579127 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 261/525] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:15:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 537 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:15:13.579646 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631da1cc-1ef2-4957-be4c-d91ca507fa9e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:13.580047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fc5605-3809-4a37-9ea3-d68e31a61c1d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:13.604887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 265/526] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:15:13 2022] POST /identity/v3/users => generated 247 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:15:13.605997 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631da1cc-1ef2-4957-be4c-d91ca507fa9e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:13.618837 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:13.766081 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fc5605-3809-4a37-9ea3-d68e31a61c1d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:13.826334 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631da1cc-1ef2-4957-be4c-d91ca507fa9e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:13.837899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20455bfe-133a-49f8-a8ab-3fbc6cdeba12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:13.839420 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20455bfe-133a-49f8-a8ab-3fbc6cdeba12 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:13.840233 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20455bfe-133a-49f8-a8ab-3fbc6cdeba12 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:13.841042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20455bfe-133a-49f8-a8ab-3fbc6cdeba12 None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:13.842765 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20455bfe-133a-49f8-a8ab-3fbc6cdeba12 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:13.843650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20455bfe-133a-49f8-a8ab-3fbc6cdeba12 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:13.848117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20455bfe-133a-49f8-a8ab-3fbc6cdeba12 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:13.885621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 262/527] 10.222.0.64 () {56 vars in 1139 bytes} [Tue Oct 4 14:15:13 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:13.887796 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20455bfe-133a-49f8-a8ab-3fbc6cdeba12 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:13.901064 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c38a1cc-3a9c-44c6-8e4d-8aaa7279322b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:13.901728 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c38a1cc-3a9c-44c6-8e4d-8aaa7279322b None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:13.902011 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c38a1cc-3a9c-44c6-8e4d-8aaa7279322b None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:13.902288 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c38a1cc-3a9c-44c6-8e4d-8aaa7279322b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:13.911291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c38a1cc-3a9c-44c6-8e4d-8aaa7279322b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:13.913685 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c38a1cc-3a9c-44c6-8e4d-8aaa7279322b None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:13.927566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-0c38a1cc-3a9c-44c6-8e4d-8aaa7279322b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:13.927566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:13.927566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:13.927566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:13.927566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:13.927566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:13.927566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:13.927566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:13.927566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:13.927566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:13.927566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:13.927566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:13.927566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:13.927566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:13.927566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:13.929426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:13.929426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:13.929426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:13.929426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:13.929426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:13.929426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:13.929426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:13.929426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:13.929426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:13.929426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:13.929426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:13.929426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:13.929426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:13.929426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:13.929426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:13.929426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:13.929426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:13.931043 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:13.931043 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:13.931043 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:13.931043 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:13.931043 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:13.931043 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:13.931043 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:13.931043 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:13.931043 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 266/528] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:13.931043 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c38a1cc-3a9c-44c6-8e4d-8aaa7279322b None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:13.992695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:14.047682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:14.085825 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c38a1cc-3a9c-44c6-8e4d-8aaa7279322b None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:14.110831 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20455bfe-133a-49f8-a8ab-3fbc6cdeba12 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:14.138703 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9582196-4a96-4bf6-b966-ea565d330f10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:14.139308 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9582196-4a96-4bf6-b966-ea565d330f10 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:14.139866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9582196-4a96-4bf6-b966-ea565d330f10 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:14.140150 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9582196-4a96-4bf6-b966-ea565d330f10 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:14.163147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d919b9-941b-44a0-89eb-6907cbafc1c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:14.163799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d919b9-941b-44a0-89eb-6907cbafc1c6 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:14.164079 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d919b9-941b-44a0-89eb-6907cbafc1c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:14.164609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d919b9-941b-44a0-89eb-6907cbafc1c6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:14.177611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d919b9-941b-44a0-89eb-6907cbafc1c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:14.179099 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d919b9-941b-44a0-89eb-6907cbafc1c6 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:14.196459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-37d919b9-941b-44a0-89eb-6907cbafc1c6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:14.196459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:14.196459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:14.196459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:14.196459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:14.196459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:14.196459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:14.196459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:14.196459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:14.196459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:14.196459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:14.196459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:14.196459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:14.196459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:14.196459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:14.196459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:14.197465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:14.197465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:14.197465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:14.197465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:14.197465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:14.197465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:14.197465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:14.197465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:14.197465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:14.197465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:14.197465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:14.197465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:14.197465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:14.197465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:14.197465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:14.197465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:14.197465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:14.198359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:14.198359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:14.198359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:14.198359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 263/529] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:14.203197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d919b9-941b-44a0-89eb-6907cbafc1c6 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:14.257483 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9582196-4a96-4bf6-b966-ea565d330f10 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:14.260172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9582196-4a96-4bf6-b966-ea565d330f10 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:14.329625 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:14.335985 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d919b9-941b-44a0-89eb-6907cbafc1c6 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:14.370075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:14.387710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9843eb83-ba5e-4adb-abf2-73fcfec09d53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:14.388368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9843eb83-ba5e-4adb-abf2-73fcfec09d53 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:14.388655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9843eb83-ba5e-4adb-abf2-73fcfec09d53 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:14.388928 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9843eb83-ba5e-4adb-abf2-73fcfec09d53 None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:14.389638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9843eb83-ba5e-4adb-abf2-73fcfec09d53 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:14.389944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9843eb83-ba5e-4adb-abf2-73fcfec09d53 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:14.391611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9843eb83-ba5e-4adb-abf2-73fcfec09d53 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:14.416463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 264/530] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:14.417478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9843eb83-ba5e-4adb-abf2-73fcfec09d53 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:14.418549 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 267/531] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:15:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 489 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:15:14.426343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9582196-4a96-4bf6-b966-ea565d330f10 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:14.573183 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9582196-4a96-4bf6-b966-ea565d330f10 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:14.574071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9843eb83-ba5e-4adb-abf2-73fcfec09d53 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:14.622637 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4669d86e-810f-45ba-87d1-943c0f964324 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:14.624001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4669d86e-810f-45ba-87d1-943c0f964324 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:14.624673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4669d86e-810f-45ba-87d1-943c0f964324 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:14.625403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4669d86e-810f-45ba-87d1-943c0f964324 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:14.627292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4669d86e-810f-45ba-87d1-943c0f964324 None admin] RBAC: Adding query filter params (role.id=3ebe3c0be8c34c5eae533311e9b44176, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=058c027ade94498fbab7b9347e87c9c3) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:14.628034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4669d86e-810f-45ba-87d1-943c0f964324 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:14.631126 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4669d86e-810f-45ba-87d1-943c0f964324 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:14.642817 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 268/532] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:15:14 2022] GET /identity/v3/role_assignments?user.id=058c027ade94498fbab7b9347e87c9c3&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=3ebe3c0be8c34c5eae533311e9b44176 => generated 259 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:14.644845 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4669d86e-810f-45ba-87d1-943c0f964324 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:14.649990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f004f739-05da-4ae2-9b36-3f09bd157d7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:14.651125 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f004f739-05da-4ae2-9b36-3f09bd157d7b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:14.651577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f004f739-05da-4ae2-9b36-3f09bd157d7b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:14.652035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f004f739-05da-4ae2-9b36-3f09bd157d7b None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:14.653628 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:14.654821 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f004f739-05da-4ae2-9b36-3f09bd157d7b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:14.655886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f004f739-05da-4ae2-9b36-3f09bd157d7b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:14.660625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f004f739-05da-4ae2-9b36-3f09bd157d7b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:14.684102 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 265/533] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:14.696456 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f004f739-05da-4ae2-9b36-3f09bd157d7b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:14.803841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:14.846446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4669d86e-810f-45ba-87d1-943c0f964324 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:14.912960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedf5da0-57b6-4282-804f-aa90bbe6f8cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:14.914482 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedf5da0-57b6-4282-804f-aa90bbe6f8cb None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:14.915185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedf5da0-57b6-4282-804f-aa90bbe6f8cb None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:14.915876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedf5da0-57b6-4282-804f-aa90bbe6f8cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:14.939987 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eedf5da0-57b6-4282-804f-aa90bbe6f8cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:14.943246 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eedf5da0-57b6-4282-804f-aa90bbe6f8cb None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:14.991543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-eedf5da0-57b6-4282-804f-aa90bbe6f8cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:14.991543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:14.991543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:14.991543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:14.991543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:14.991543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:14.991543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:14.991543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:14.991543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:14.991543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:14.991543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:14.991543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:14.991543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:14.991543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:14.991543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:14.991543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:14.993237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:14.993237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:14.993237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:14.993237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:14.993237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:14.993237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:14.993237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:14.993237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:14.993237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:14.993237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:14.993237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:14.993237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:14.993237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:14.993237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:14.993237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:14.993237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:14.993237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:14.995204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:14.995204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:14.995204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:14.999263 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f004f739-05da-4ae2-9b36-3f09bd157d7b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:15.000962 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 269/534] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 357 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:15.010124 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedf5da0-57b6-4282-804f-aa90bbe6f8cb None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:15.163924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb66379-d231-41c2-b4b5-11f0be0f27ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:15.164992 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb66379-d231-41c2-b4b5-11f0be0f27ec None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:15.165502 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb66379-d231-41c2-b4b5-11f0be0f27ec None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:15.173216 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb66379-d231-41c2-b4b5-11f0be0f27ec None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:15.196628 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb66379-d231-41c2-b4b5-11f0be0f27ec None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:15.199387 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb66379-d231-41c2-b4b5-11f0be0f27ec None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:15.226519 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:15.228237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-3bb66379-d231-41c2-b4b5-11f0be0f27ec None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:15.228237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:15.228237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:15.228237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:15.228237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:15.228237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:15.228237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:15.228237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:15.228237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:15.228237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:15.228237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:15.228237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:15.228237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:15.228237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:15.228237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:15.230323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:15.230323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:15.230323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:15.230323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:15.230323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:15.230323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:15.230323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:15.230323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:15.230323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:15.230323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:15.230323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:15.230323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:15.230323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:15.230323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:15.230323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:15.230323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:15.230323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:15.232010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:15.232010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:15.232010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:15.232010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:15.232010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:15.232010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:15.232010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:15.232010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:15.233644 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 266/535] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 537 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:15.239806 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb66379-d231-41c2-b4b5-11f0be0f27ec None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:15.331890 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedf5da0-57b6-4282-804f-aa90bbe6f8cb None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:15.398308 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7147762a-347a-456e-8676-96a215fdff35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:15.398944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7147762a-347a-456e-8676-96a215fdff35 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:15.399228 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7147762a-347a-456e-8676-96a215fdff35 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:15.399569 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7147762a-347a-456e-8676-96a215fdff35 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:15.400250 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7147762a-347a-456e-8676-96a215fdff35 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:15.400612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7147762a-347a-456e-8676-96a215fdff35 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:15.402070 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7147762a-347a-456e-8676-96a215fdff35 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:15.435305 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 270/536] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 422 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:15.439163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7147762a-347a-456e-8676-96a215fdff35 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:15.501039 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb66379-d231-41c2-b4b5-11f0be0f27ec None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:15.508554 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:15.591631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c2987b-cc18-4c9c-ace6-c67cf7a69c97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:15.592680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c2987b-cc18-4c9c-ace6-c67cf7a69c97 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:15.593170 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c2987b-cc18-4c9c-ace6-c67cf7a69c97 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:15.593646 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c2987b-cc18-4c9c-ace6-c67cf7a69c97 None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:15.594681 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c2987b-cc18-4c9c-ace6-c67cf7a69c97 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:15.595223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c2987b-cc18-4c9c-ace6-c67cf7a69c97 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:15.597011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c2987b-cc18-4c9c-ace6-c67cf7a69c97 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:15.616709 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 267/537] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 377 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:15:15.619081 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c2987b-cc18-4c9c-ace6-c67cf7a69c97 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:15.678397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7147762a-347a-456e-8676-96a215fdff35 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:15.723264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:15.745017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c2987b-cc18-4c9c-ace6-c67cf7a69c97 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:15.787054 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84c24ba-5394-431e-8a9a-bfbd949dd7e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:15.787848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84c24ba-5394-431e-8a9a-bfbd949dd7e6 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:15.788225 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84c24ba-5394-431e-8a9a-bfbd949dd7e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:15.788631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84c24ba-5394-431e-8a9a-bfbd949dd7e6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:15.789481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84c24ba-5394-431e-8a9a-bfbd949dd7e6 None admin] RBAC: Adding query filter params (role.id=3ebe3c0be8c34c5eae533311e9b44176, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=03081d7d581745c88c12e9d089fcb2d3) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:15.789915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84c24ba-5394-431e-8a9a-bfbd949dd7e6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:15.791589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84c24ba-5394-431e-8a9a-bfbd949dd7e6 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:15.797106 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ef47f9-09a7-43f9-82ea-6d8a8e690502 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:15.798305 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ef47f9-09a7-43f9-82ea-6d8a8e690502 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:15.798607 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ef47f9-09a7-43f9-82ea-6d8a8e690502 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:15.798884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ef47f9-09a7-43f9-82ea-6d8a8e690502 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:15.802987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 268/538] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:15:15 2022] GET /identity/v3/role_assignments?user.id=03081d7d581745c88c12e9d089fcb2d3&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=3ebe3c0be8c34c5eae533311e9b44176 => generated 259 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:15.821814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ef47f9-09a7-43f9-82ea-6d8a8e690502 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:15.823385 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ef47f9-09a7-43f9-82ea-6d8a8e690502 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:15.847291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-c3ef47f9-09a7-43f9-82ea-6d8a8e690502 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:15.847291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:15.847291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:15.847291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:15.847291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:15.847291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:15.847291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:15.847291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:15.847291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:15.847291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:15.847291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:15.847291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:15.847291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:15.847291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:15.847291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:15.849117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:15.849117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:15.849117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:15.849117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:15.849117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:15.849117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:15.849117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:15.849117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:15.849117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:15.849117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:15.849117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:15.849117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:15.849117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:15.849117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:15.849117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:15.849117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:15.849117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:15.852518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:15.852518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:15.852518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:15.852518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:15.852518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:15.852518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:15.852518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:15.852518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:15.852518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 271/539] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 410 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:15.860345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84c24ba-5394-431e-8a9a-bfbd949dd7e6 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:15.922177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:16.100829 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84c24ba-5394-431e-8a9a-bfbd949dd7e6 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:16.152348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b22d401-7e42-4a1c-b458-12dec3951a1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:16.152970 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b22d401-7e42-4a1c-b458-12dec3951a1a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:16.153249 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b22d401-7e42-4a1c-b458-12dec3951a1a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:16.153523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b22d401-7e42-4a1c-b458-12dec3951a1a None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:16.154226 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b22d401-7e42-4a1c-b458-12dec3951a1a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:16.154529 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b22d401-7e42-4a1c-b458-12dec3951a1a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:16.156256 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b22d401-7e42-4a1c-b458-12dec3951a1a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:16.173963 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 269/540] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:16.181595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ef47f9-09a7-43f9-82ea-6d8a8e690502 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:16.421064 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ef47f9-09a7-43f9-82ea-6d8a8e690502 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:16.523098 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df42212-8a33-4fbc-81f6-ced2845fd690 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:16.524100 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df42212-8a33-4fbc-81f6-ced2845fd690 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:16.524605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df42212-8a33-4fbc-81f6-ced2845fd690 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:16.525078 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df42212-8a33-4fbc-81f6-ced2845fd690 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:16.526415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df42212-8a33-4fbc-81f6-ced2845fd690 None admin] RBAC: Adding query filter params (role.id=3ebe3c0be8c34c5eae533311e9b44176, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=8dbeccfb9a194992959f0b341ef0d4a5) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:16.526946 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df42212-8a33-4fbc-81f6-ced2845fd690 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:16.529746 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df42212-8a33-4fbc-81f6-ced2845fd690 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:16.547894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 272/541] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:15:16 2022] GET /identity/v3/role_assignments?user.id=8dbeccfb9a194992959f0b341ef0d4a5&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=3ebe3c0be8c34c5eae533311e9b44176 => generated 259 bytes in 367 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:17.100968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65bebbb-7d93-4e58-83b5-23e2c155552f None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:17.101399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65bebbb-7d93-4e58-83b5-23e2c155552f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:17.101551 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65bebbb-7d93-4e58-83b5-23e2c155552f None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:17.103307 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 270/542] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:17.108593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed00df5-ace0-481b-a525-e085957f9a34 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:17.108880 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed00df5-ace0-481b-a525-e085957f9a34 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:17.109223 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed00df5-ace0-481b-a525-e085957f9a34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:17.140249 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-fed00df5-ace0-481b-a525-e085957f9a34 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:17.142192 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-fed00df5-ace0-481b-a525-e085957f9a34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:17.199767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:17.263025 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 273/543] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:17.302115 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f8e073-529c-44dd-8bcf-03d2fd11d261 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:17.302976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f8e073-529c-44dd-8bcf-03d2fd11d261 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:17.302976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f8e073-529c-44dd-8bcf-03d2fd11d261 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:17.304511 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 271/544] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:15:17.310927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e955564d-9c30-4ff1-81af-39aebc3a6ffa None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:17.311204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e955564d-9c30-4ff1-81af-39aebc3a6ffa None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:17.311553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e955564d-9c30-4ff1-81af-39aebc3a6ffa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:17.341491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-e955564d-9c30-4ff1-81af-39aebc3a6ffa None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:17.342738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-e955564d-9c30-4ff1-81af-39aebc3a6ffa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:17.454205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 274/545] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:17.693306 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f8e073-529c-44dd-8bcf-03d2fd11d261 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:17.807173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e955564d-9c30-4ff1-81af-39aebc3a6ffa None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:17.810096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:17.838211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f8e073-529c-44dd-8bcf-03d2fd11d261 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:17.953006 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007a1f15-04a9-41b3-be15-2b8ebd749210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:17.954081 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007a1f15-04a9-41b3-be15-2b8ebd749210 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:17.957868 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007a1f15-04a9-41b3-be15-2b8ebd749210 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:17.957868 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007a1f15-04a9-41b3-be15-2b8ebd749210 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:18.010221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:18.018997 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007a1f15-04a9-41b3-be15-2b8ebd749210 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:18.042374 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e955564d-9c30-4ff1-81af-39aebc3a6ffa None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:18.049705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-007a1f15-04a9-41b3-be15-2b8ebd749210 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:18.049705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:18.049705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:18.049705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:18.049705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:18.049705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:18.049705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:18.049705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:18.049705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:18.049705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:18.049705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:18.049705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:18.049705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:18.049705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:18.049705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:18.049705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:18.050752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:18.050752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:18.050752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:18.050752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:18.050752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:18.050752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:18.050752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:18.050752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:18.050752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:18.050752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:18.050752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:18.050752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:18.050752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:18.050752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:18.050752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:18.050752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:18.051613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:18.051613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:18.051613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:18.051613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:18.051613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:18.051613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:18.051613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:18.051613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:18.051613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:18.051613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:18.051613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:18.051613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:18.060938 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 272/546] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:17 2022] GET /identity/v3/roles/service => generated 85 bytes in 368 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:18.062336 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007a1f15-04a9-41b3-be15-2b8ebd749210 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:18.189172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f37f3f4-b293-47ff-adbb-76b68fc939d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:18.190176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f37f3f4-b293-47ff-adbb-76b68fc939d6 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:18.190645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f37f3f4-b293-47ff-adbb-76b68fc939d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:18.191125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f37f3f4-b293-47ff-adbb-76b68fc939d6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:18.241551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f37f3f4-b293-47ff-adbb-76b68fc939d6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:18.251918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-0f37f3f4-b293-47ff-adbb-76b68fc939d6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:18.251918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:18.251918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:18.251918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:18.251918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:18.251918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:18.251918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:18.251918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:18.251918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:18.251918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:18.251918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:18.251918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:18.251918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:18.251918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:18.251918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:18.251918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:18.253004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:18.253004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:18.253004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:18.253004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:18.253004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:18.253004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:18.253004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:18.253004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:18.253004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:18.253004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:18.253004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:18.253004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:18.253004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:18.253004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:18.253004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:18.253004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:18.254292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:18.254292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:18.254292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:18.254292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:18.254292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:18.254292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:18.254292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:18.254292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:18.254292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:18.254292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:18.254292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:18.254292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:18.264992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 275/547] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:17 2022] GET /identity/v3/roles/service => generated 85 bytes in 458 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:18.275662 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f37f3f4-b293-47ff-adbb-76b68fc939d6 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:18.492483 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007a1f15-04a9-41b3-be15-2b8ebd749210 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:18.555256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f37f3f4-b293-47ff-adbb-76b68fc939d6 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:18.596759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9238ea88-3cbb-4c8e-a161-95d8c1484482 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:18.597754 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9238ea88-3cbb-4c8e-a161-95d8c1484482 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:18.598220 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9238ea88-3cbb-4c8e-a161-95d8c1484482 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:18.598683 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9238ea88-3cbb-4c8e-a161-95d8c1484482 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:18.600151 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9238ea88-3cbb-4c8e-a161-95d8c1484482 None admin] RBAC: Adding query filter params (name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:18.600889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9238ea88-3cbb-4c8e-a161-95d8c1484482 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:18.603316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9238ea88-3cbb-4c8e-a161-95d8c1484482 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:18.648240 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 273/548] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:15:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 586 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:18.651106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf551afd-939b-4db5-b749-29a5f53ae104 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:18.651607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf551afd-939b-4db5-b749-29a5f53ae104 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:18.652069 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf551afd-939b-4db5-b749-29a5f53ae104 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:18.664398 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 274/549] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:18 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:18.664897 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf551afd-939b-4db5-b749-29a5f53ae104 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:18.698777 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bbaeb0-8d82-465f-aa07-7ceda4a9dd0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:18.699765 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bbaeb0-8d82-465f-aa07-7ceda4a9dd0e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:18.700240 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bbaeb0-8d82-465f-aa07-7ceda4a9dd0e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:18.706593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bbaeb0-8d82-465f-aa07-7ceda4a9dd0e None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:18.707768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bbaeb0-8d82-465f-aa07-7ceda4a9dd0e None admin] RBAC: Adding query filter params (name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:18.708281 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bbaeb0-8d82-465f-aa07-7ceda4a9dd0e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:18.710729 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bbaeb0-8d82-465f-aa07-7ceda4a9dd0e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:18.717007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:18.752608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 276/550] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:15:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 474 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:15:18.752608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939e7621-0d7a-42aa-bdae-9f17808a6579 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:18.752608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939e7621-0d7a-42aa-bdae-9f17808a6579 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:18.752608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939e7621-0d7a-42aa-bdae-9f17808a6579 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:18.821158 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-939e7621-0d7a-42aa-bdae-9f17808a6579 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:18.823541 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-939e7621-0d7a-42aa-bdae-9f17808a6579 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:18.871157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:18.942962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf551afd-939b-4db5-b749-29a5f53ae104 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:18.977192 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 277/551] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:18.979310 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939e7621-0d7a-42aa-bdae-9f17808a6579 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:19.029659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca179f19-d918-4f44-aabb-6d2f85be5023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:19.031011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca179f19-d918-4f44-aabb-6d2f85be5023 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:19.032123 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca179f19-d918-4f44-aabb-6d2f85be5023 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:19.032634 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca179f19-d918-4f44-aabb-6d2f85be5023 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:19.050060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca179f19-d918-4f44-aabb-6d2f85be5023 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:19.051937 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca179f19-d918-4f44-aabb-6d2f85be5023 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:19.075695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-ca179f19-d918-4f44-aabb-6d2f85be5023 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:19.075695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:19.075695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:19.075695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:19.075695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:19.075695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:19.075695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:19.075695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:19.075695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:19.075695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:19.075695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:19.075695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:19.075695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:19.075695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:19.075695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:19.075695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:19.077595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:19.077595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:19.077595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:19.077595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:19.077595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:19.077595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:19.077595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:19.077595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:19.077595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:19.077595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:19.077595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:19.077595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:19.077595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:19.077595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:19.077595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:19.077595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:19.077595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:19.079584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:19.079584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:19.079584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:19.083177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 275/552] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 419 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:19.085867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bd0e34-b99a-4734-92ce-c51108ba5205 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:19.086728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bd0e34-b99a-4734-92ce-c51108ba5205 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:19.087498 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bd0e34-b99a-4734-92ce-c51108ba5205 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:19.096076 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 276/553] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:19 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:19.098039 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bd0e34-b99a-4734-92ce-c51108ba5205 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:19.154514 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939e7621-0d7a-42aa-bdae-9f17808a6579 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:19.221721 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:19.233799 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875a86a1-b6fa-40ea-a804-c9f8a447aaf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:19.235364 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875a86a1-b6fa-40ea-a804-c9f8a447aaf1 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:19.236004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875a86a1-b6fa-40ea-a804-c9f8a447aaf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:19.236506 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875a86a1-b6fa-40ea-a804-c9f8a447aaf1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:19.261365 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875a86a1-b6fa-40ea-a804-c9f8a447aaf1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:19.264980 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875a86a1-b6fa-40ea-a804-c9f8a447aaf1 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:19.300208 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bd0e34-b99a-4734-92ce-c51108ba5205 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:19.300555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-875a86a1-b6fa-40ea-a804-c9f8a447aaf1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:19.300555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:19.300555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:19.300555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:19.300555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:19.300555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:19.300555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:19.300555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:19.300555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:19.300555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:19.300555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:19.300555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:19.300555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:19.300555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:19.300555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:19.300555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:19.302231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:19.302231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:19.302231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:19.302231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:19.302231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:19.302231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:19.302231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:19.302231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:19.302231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:19.302231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:19.302231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:19.302231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:19.302231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:19.302231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:19.302231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:19.302231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:19.302231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:19.304069 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:19.304069 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:19.304069 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:19.304069 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 278/554] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 324 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:19.305476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fd873f-45b2-48c1-aad0-5bb92bea422b None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:19.305476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fd873f-45b2-48c1-aad0-5bb92bea422b None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:19.305476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fd873f-45b2-48c1-aad0-5bb92bea422b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:19.318770 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:19.339118 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-86fd873f-45b2-48c1-aad0-5bb92bea422b None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:19.341065 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-86fd873f-45b2-48c1-aad0-5bb92bea422b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:19.364778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c59eea8-20ba-4e3c-b398-57134339d948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:19.365765 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c59eea8-20ba-4e3c-b398-57134339d948 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:19.366230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c59eea8-20ba-4e3c-b398-57134339d948 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:19.366652 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c59eea8-20ba-4e3c-b398-57134339d948 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:19.367776 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c59eea8-20ba-4e3c-b398-57134339d948 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:19.368319 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c59eea8-20ba-4e3c-b398-57134339d948 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:19.370778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c59eea8-20ba-4e3c-b398-57134339d948 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:19.396726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 277/555] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:19.398259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c59eea8-20ba-4e3c-b398-57134339d948 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:19.446734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 279/556] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:19.454066 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fd873f-45b2-48c1-aad0-5bb92bea422b None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:19.586856 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c59eea8-20ba-4e3c-b398-57134339d948 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:19.622751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:19.640719 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c56daaf-48f3-4d6f-867a-339916a316d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:19.641921 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c56daaf-48f3-4d6f-867a-339916a316d0 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:19.642197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c56daaf-48f3-4d6f-867a-339916a316d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:19.642474 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c56daaf-48f3-4d6f-867a-339916a316d0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:19.647894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fd873f-45b2-48c1-aad0-5bb92bea422b None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:19.657671 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c56daaf-48f3-4d6f-867a-339916a316d0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:19.662756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-2c56daaf-48f3-4d6f-867a-339916a316d0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:19.662756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:19.662756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:19.662756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:19.662756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:19.662756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:19.662756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:19.662756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:19.662756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:19.662756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:19.662756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:19.662756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:19.662756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:19.662756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:19.662756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:19.662756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:19.663968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:19.663968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:19.663968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:19.663968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:19.663968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:19.663968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:19.663968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:19.663968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:19.663968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:19.663968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:19.663968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:19.663968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:19.663968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:19.663968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:19.663968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:19.663968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:19.665019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:19.665019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:19.665019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:19.665019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:19.665019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:19.665019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:19.665019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:19.665019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:19.665019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:19.665019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:19.665019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:19.665019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:19.665019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 278/557] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:19.665715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c56daaf-48f3-4d6f-867a-339916a316d0 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:19.705398 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0836a202-f5c4-4fe1-aca0-d6d71cd2d2cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:19.706435 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0836a202-f5c4-4fe1-aca0-d6d71cd2d2cf None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:19.706900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0836a202-f5c4-4fe1-aca0-d6d71cd2d2cf None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:19.707347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0836a202-f5c4-4fe1-aca0-d6d71cd2d2cf None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:19.708636 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0836a202-f5c4-4fe1-aca0-d6d71cd2d2cf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:19.709215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0836a202-f5c4-4fe1-aca0-d6d71cd2d2cf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:19.711654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0836a202-f5c4-4fe1-aca0-d6d71cd2d2cf None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:19.714021 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:19.737264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 280/558] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:19.737264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1485ae83-3ac2-4c99-8d9b-68fe232f4f32 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:19.737264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1485ae83-3ac2-4c99-8d9b-68fe232f4f32 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:19.737264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1485ae83-3ac2-4c99-8d9b-68fe232f4f32 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:19.737264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 281/559] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:19.740076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1485ae83-3ac2-4c99-8d9b-68fe232f4f32 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:19.848076 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c56daaf-48f3-4d6f-867a-339916a316d0 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:19.903963 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108c7814-d35f-45f8-a3e5-1bbbc46b912a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:19.904349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1485ae83-3ac2-4c99-8d9b-68fe232f4f32 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:19.904750 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108c7814-d35f-45f8-a3e5-1bbbc46b912a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:19.905074 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108c7814-d35f-45f8-a3e5-1bbbc46b912a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:19.905655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108c7814-d35f-45f8-a3e5-1bbbc46b912a None admin] PATH_INFO: `/v3/users/glance` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:19.914441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108c7814-d35f-45f8-a3e5-1bbbc46b912a None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:19.916078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108c7814-d35f-45f8-a3e5-1bbbc46b912a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:19.929655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-108c7814-d35f-45f8-a3e5-1bbbc46b912a None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 04 14:15:19.929655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:19.929655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:19.929655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:19.929655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:19.929655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:19.929655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:19.929655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:19.929655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:19.929655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:19.929655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:19.929655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:19.929655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:19.929655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:19.929655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:19.929655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:19.931489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:19.933118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:19.933118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:19.933118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:19.933118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:19.933118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:19.933118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:19.933118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:19.933118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:19.933118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 04 14:15:19.933118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:19.933118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 279/560] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:15:19 2022] GET /identity/v3/users/glance => generated 84 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:19.933118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108c7814-d35f-45f8-a3e5-1bbbc46b912a None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:19.952477 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc037c0-cc4e-4574-8366-a89255980058 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:19.953106 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc037c0-cc4e-4574-8366-a89255980058 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:19.953411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc037c0-cc4e-4574-8366-a89255980058 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:19.954212 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc037c0-cc4e-4574-8366-a89255980058 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:19.954908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc037c0-cc4e-4574-8366-a89255980058 None admin] RBAC: Adding query filter params (name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:19.955215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc037c0-cc4e-4574-8366-a89255980058 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:19.956683 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc037c0-cc4e-4574-8366-a89255980058 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:19.966165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 282/561] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:15:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:19.968870 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8506d037-fc26-4ded-8cce-55193d75fde2 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:19.969898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8506d037-fc26-4ded-8cce-55193d75fde2 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:19.970391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8506d037-fc26-4ded-8cce-55193d75fde2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:19.985420 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:19.993163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-8506d037-fc26-4ded-8cce-55193d75fde2 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:19.994407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8506d037-fc26-4ded-8cce-55193d75fde2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:20.052875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108c7814-d35f-45f8-a3e5-1bbbc46b912a None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:20.053186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:20.072144 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 283/562] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:20.073297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8506d037-fc26-4ded-8cce-55193d75fde2 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:20.101787 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3148e77a-39ba-4f96-86da-ad8934560871 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:20.102751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3148e77a-39ba-4f96-86da-ad8934560871 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:20.103348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3148e77a-39ba-4f96-86da-ad8934560871 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:20.103836 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3148e77a-39ba-4f96-86da-ad8934560871 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:20.118799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3148e77a-39ba-4f96-86da-ad8934560871 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:20.121414 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3148e77a-39ba-4f96-86da-ad8934560871 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:20.144296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-3148e77a-39ba-4f96-86da-ad8934560871 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 14:15:20.144296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:20.144296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:20.144296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:20.144296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:20.144296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:20.144296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:20.144296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:20.144296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:20.144296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:20.144296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:20.144296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:20.144296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:20.144296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:20.144296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:20.144296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:20.145380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:20.151463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:20.151463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:20.151463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:20.151463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:20.151463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:20.151463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:20.151463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:20.151463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:20.151463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 14:15:20.151463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:20.151463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 280/563] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:15:19 2022] GET /identity/v3/users/swift => generated 83 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:20.151463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3148e77a-39ba-4f96-86da-ad8934560871 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:20.240371 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8506d037-fc26-4ded-8cce-55193d75fde2 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:20.301065 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149c0c95-6f89-4067-89aa-a59158074cee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:20.302060 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149c0c95-6f89-4067-89aa-a59158074cee None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:20.302502 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149c0c95-6f89-4067-89aa-a59158074cee None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:20.302884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149c0c95-6f89-4067-89aa-a59158074cee None admin] PATH_INFO: `/v3/roles/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:20.319925 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149c0c95-6f89-4067-89aa-a59158074cee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:20.329277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-149c0c95-6f89-4067-89aa-a59158074cee None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:20.329277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:20.329277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:20.329277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:20.329277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:20.329277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:20.329277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:20.329277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:20.329277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:20.329277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:20.329277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:20.329277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:20.329277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:20.329277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:20.329277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:20.329277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:20.331218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:20.331218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:20.331218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:20.331218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:20.331218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:20.331218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:20.331218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:20.331218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:20.331218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:20.331218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:20.331218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:20.331218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:20.331218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:20.331218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:20.331218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:20.331218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:20.333119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:20.333119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:20.333119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:20.333119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:20.333119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:20.333119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:20.333119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:20.333119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:20.333119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:20.333119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:20.333119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:20.333119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:20.333119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 284/564] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:20.334457 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149c0c95-6f89-4067-89aa-a59158074cee None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:20.338404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3148e77a-39ba-4f96-86da-ad8934560871 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:20.341258 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:20.385680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249a79e6-c7e5-4de9-90b7-9b3e486ac7d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:20.386298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249a79e6-c7e5-4de9-90b7-9b3e486ac7d1 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:20.386575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249a79e6-c7e5-4de9-90b7-9b3e486ac7d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:20.386874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249a79e6-c7e5-4de9-90b7-9b3e486ac7d1 None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:20.387635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249a79e6-c7e5-4de9-90b7-9b3e486ac7d1 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:20.387936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249a79e6-c7e5-4de9-90b7-9b3e486ac7d1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:20.389511 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249a79e6-c7e5-4de9-90b7-9b3e486ac7d1 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:20.394256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:20.412930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 281/565] 10.222.0.64 () {56 vars in 1137 bytes} [Tue Oct 4 14:15:20 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:20.414341 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249a79e6-c7e5-4de9-90b7-9b3e486ac7d1 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:20.462922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149c0c95-6f89-4067-89aa-a59158074cee None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:20.501730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ef3c5c-7064-409f-8541-1ffa6d5f0789 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:20.502376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ef3c5c-7064-409f-8541-1ffa6d5f0789 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:20.502659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ef3c5c-7064-409f-8541-1ffa6d5f0789 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:20.502938 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ef3c5c-7064-409f-8541-1ffa6d5f0789 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:20.514887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ef3c5c-7064-409f-8541-1ffa6d5f0789 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:20.517229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ef3c5c-7064-409f-8541-1ffa6d5f0789 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:20.537193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-66ef3c5c-7064-409f-8541-1ffa6d5f0789 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:20.537193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:20.537193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:20.537193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:20.537193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:20.537193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:20.537193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:20.537193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:20.537193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:20.537193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:20.537193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:20.537193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:20.537193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:20.537193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:20.537193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:20.537193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:20.539148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:20.539148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:20.539148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:20.539148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:20.539148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:20.539148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:20.539148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:20.539148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:20.539148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:20.539148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:20.539148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:20.539148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:20.539148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:20.539148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:20.539148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:20.539148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:20.539148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:20.541544 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:20.541544 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:20.541544 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:20.541544 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 285/566] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:20.541544 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ef3c5c-7064-409f-8541-1ffa6d5f0789 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:20.542112 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249a79e6-c7e5-4de9-90b7-9b3e486ac7d1 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:20.582296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2e424a-bee2-47fb-a077-10db1ca0f794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:20.583212 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2e424a-bee2-47fb-a077-10db1ca0f794 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:20.583487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2e424a-bee2-47fb-a077-10db1ca0f794 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:20.583759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2e424a-bee2-47fb-a077-10db1ca0f794 None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:20.584554 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2e424a-bee2-47fb-a077-10db1ca0f794 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:20.584861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2e424a-bee2-47fb-a077-10db1ca0f794 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:20.586316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2e424a-bee2-47fb-a077-10db1ca0f794 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:20.606390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 282/567] 10.222.0.64 () {56 vars in 1135 bytes} [Tue Oct 4 14:15:20 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:20.607373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2e424a-bee2-47fb-a077-10db1ca0f794 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:20.630571 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:20.663753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:20.668840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ef3c5c-7064-409f-8541-1ffa6d5f0789 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:20.721019 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b4ee3c-edc0-46b3-b8cb-ff6ed2ee8822 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:20.721696 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b4ee3c-edc0-46b3-b8cb-ff6ed2ee8822 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:20.721985 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b4ee3c-edc0-46b3-b8cb-ff6ed2ee8822 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:20.722262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b4ee3c-edc0-46b3-b8cb-ff6ed2ee8822 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:20.722972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b4ee3c-edc0-46b3-b8cb-ff6ed2ee8822 None admin] RBAC: Adding query filter params (name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:20.723278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b4ee3c-edc0-46b3-b8cb-ff6ed2ee8822 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:20.724733 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b4ee3c-edc0-46b3-b8cb-ff6ed2ee8822 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:20.733555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 286/568] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:15:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:20.734424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b4ee3c-edc0-46b3-b8cb-ff6ed2ee8822 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:20.743760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2e424a-bee2-47fb-a077-10db1ca0f794 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:20.783898 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ec5346-8386-4c42-a491-052cf99bbbf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:20.784591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ec5346-8386-4c42-a491-052cf99bbbf9 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:20.784866 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ec5346-8386-4c42-a491-052cf99bbbf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:20.785140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ec5346-8386-4c42-a491-052cf99bbbf9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:20.797796 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ec5346-8386-4c42-a491-052cf99bbbf9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:20.802757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-32ec5346-8386-4c42-a491-052cf99bbbf9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:20.802757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:20.802757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:20.802757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:20.802757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:20.802757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:20.802757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:20.802757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:20.802757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:20.802757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:20.802757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:20.802757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:20.802757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:20.802757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:20.802757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:20.802757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:20.804660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:20.804660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:20.804660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:20.804660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:20.804660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:20.804660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:20.804660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:20.804660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:20.804660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:20.804660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:20.804660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:20.804660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:20.804660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:20.804660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:20.804660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:20.804660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:20.806354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:20.806354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:20.806354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:20.806354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:20.806354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:20.806354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:20.806354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:20.806354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:20.806354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:20.806354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:20.806354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:20.806354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:20.806354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 283/569] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:20.806354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ec5346-8386-4c42-a491-052cf99bbbf9 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:20.864513 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b4ee3c-edc0-46b3-b8cb-ff6ed2ee8822 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:20.903715 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ae4555-e958-4fd2-a04c-1d57f70a2d81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:20.904573 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ae4555-e958-4fd2-a04c-1d57f70a2d81 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:20.904890 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ae4555-e958-4fd2-a04c-1d57f70a2d81 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:20.905435 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ae4555-e958-4fd2-a04c-1d57f70a2d81 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:20.907734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:20.918996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ae4555-e958-4fd2-a04c-1d57f70a2d81 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:20.921025 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ae4555-e958-4fd2-a04c-1d57f70a2d81 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:20.932614 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ec5346-8386-4c42-a491-052cf99bbbf9 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:20.936682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-b6ae4555-e958-4fd2-a04c-1d57f70a2d81 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:20.936682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:20.936682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:20.936682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:20.936682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:20.936682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:20.936682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:20.936682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:20.936682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:20.936682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:20.936682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:20.936682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:20.936682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:20.936682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:20.936682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:20.936682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:20.939726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:20.939726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:20.939726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:20.939726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:20.939726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:20.939726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:20.939726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:20.939726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:20.939726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:20.939726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:20.939726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:20.939726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:20.939726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:20.939726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:20.939726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:20.939726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:20.939726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:20.941511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:20.941511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:20.941511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:20.941511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 287/570] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:20.941511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ae4555-e958-4fd2-a04c-1d57f70a2d81 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:20.956668 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:20.976331 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8e17a9-9f65-4dee-aac5-7953ff11dfba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:20.976964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8e17a9-9f65-4dee-aac5-7953ff11dfba None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:20.977243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8e17a9-9f65-4dee-aac5-7953ff11dfba None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:20.977514 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8e17a9-9f65-4dee-aac5-7953ff11dfba None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:20.978181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8e17a9-9f65-4dee-aac5-7953ff11dfba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:20.978484 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8e17a9-9f65-4dee-aac5-7953ff11dfba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:20.979885 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8e17a9-9f65-4dee-aac5-7953ff11dfba None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:20.993990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 284/571] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:20.994859 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8e17a9-9f65-4dee-aac5-7953ff11dfba None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:21.058850 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ae4555-e958-4fd2-a04c-1d57f70a2d81 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:21.097626 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a443580-c8eb-4812-9773-ac95c7583b14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:21.098258 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a443580-c8eb-4812-9773-ac95c7583b14 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:21.098535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a443580-c8eb-4812-9773-ac95c7583b14 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:21.098800 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a443580-c8eb-4812-9773-ac95c7583b14 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:21.108565 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8e17a9-9f65-4dee-aac5-7953ff11dfba None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:21.148219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ece29d-f19e-4b66-ba41-e5c094302cc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:21.148900 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ece29d-f19e-4b66-ba41-e5c094302cc3 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:21.149175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ece29d-f19e-4b66-ba41-e5c094302cc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:21.149456 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ece29d-f19e-4b66-ba41-e5c094302cc3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:21.157901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:21.163429 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ece29d-f19e-4b66-ba41-e5c094302cc3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:21.165479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ece29d-f19e-4b66-ba41-e5c094302cc3 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:21.182381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-a5ece29d-f19e-4b66-ba41-e5c094302cc3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:21.182381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:21.182381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:21.182381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:21.182381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:21.182381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:21.182381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:21.182381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:21.182381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:21.182381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:21.182381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:21.182381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:21.182381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:21.182381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:21.182381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:21.182381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:21.183399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:21.183399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:21.183399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:21.183399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:21.183399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:21.183399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:21.183399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:21.183399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:21.183399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:21.183399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:21.183399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:21.183399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:21.183399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:21.183399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:21.183399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:21.183399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:21.183399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:21.184621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:21.184621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:21.184621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:21.184621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 285/572] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:21.184621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ece29d-f19e-4b66-ba41-e5c094302cc3 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:21.185008 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a443580-c8eb-4812-9773-ac95c7583b14 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:21.187102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a443580-c8eb-4812-9773-ac95c7583b14 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:21.209223 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:21.309638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 288/573] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:15:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:15:21.310509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a443580-c8eb-4812-9773-ac95c7583b14 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:21.325544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ece29d-f19e-4b66-ba41-e5c094302cc3 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:21.365413 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9891bc18-cdb6-4556-98ec-290b354db20e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:21.366162 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9891bc18-cdb6-4556-98ec-290b354db20e None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:21.366495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9891bc18-cdb6-4556-98ec-290b354db20e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:21.366870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9891bc18-cdb6-4556-98ec-290b354db20e None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:21.367630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9891bc18-cdb6-4556-98ec-290b354db20e None admin] RBAC: Adding query filter params (name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:21.368059 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9891bc18-cdb6-4556-98ec-290b354db20e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:21.369729 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9891bc18-cdb6-4556-98ec-290b354db20e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:21.380477 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 286/574] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:15:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:21.382249 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9891bc18-cdb6-4556-98ec-290b354db20e None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:21.439933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a443580-c8eb-4812-9773-ac95c7583b14 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:21.451424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:21.457953 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:21.488483 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e248fb16-056b-4c52-bd98-2d92393b828e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:21.489189 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e248fb16-056b-4c52-bd98-2d92393b828e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:21.489548 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e248fb16-056b-4c52-bd98-2d92393b828e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:21.489946 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e248fb16-056b-4c52-bd98-2d92393b828e None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:21.490765 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e248fb16-056b-4c52-bd98-2d92393b828e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:21.491135 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e248fb16-056b-4c52-bd98-2d92393b828e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:21.492645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e248fb16-056b-4c52-bd98-2d92393b828e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:21.506832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 289/575] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:21.507969 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e248fb16-056b-4c52-bd98-2d92393b828e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:21.512029 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9891bc18-cdb6-4556-98ec-290b354db20e None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:21.556521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff4a037-d194-427d-a025-abfd90cfacdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:21.557765 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff4a037-d194-427d-a025-abfd90cfacdf None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:21.558211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff4a037-d194-427d-a025-abfd90cfacdf None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:21.558624 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff4a037-d194-427d-a025-abfd90cfacdf None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:21.573398 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff4a037-d194-427d-a025-abfd90cfacdf None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:21.576038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff4a037-d194-427d-a025-abfd90cfacdf None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:21.598353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-6ff4a037-d194-427d-a025-abfd90cfacdf None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 14:15:21.598353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:21.598353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:21.598353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:21.598353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:21.598353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:21.598353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:21.598353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:21.598353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:21.598353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:21.598353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:21.598353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:21.598353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:21.598353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:21.598353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:21.598353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:21.599591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:21.600805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:21.600805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:21.600805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:21.600805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:21.600805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:21.600805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:21.600805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:21.600805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:21.600805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 14:15:21.600805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:21.607119 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 287/576] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:15:21 2022] GET /identity/v3/users/cinder => generated 84 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:21.608652 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff4a037-d194-427d-a025-abfd90cfacdf None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:21.638566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e248fb16-056b-4c52-bd98-2d92393b828e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:21.676646 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2953560a-276f-4451-9ea4-3b017294490e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:21.677515 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2953560a-276f-4451-9ea4-3b017294490e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:21.677623 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2953560a-276f-4451-9ea4-3b017294490e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:21.677877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2953560a-276f-4451-9ea4-3b017294490e None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:21.678542 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2953560a-276f-4451-9ea4-3b017294490e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:21.678853 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2953560a-276f-4451-9ea4-3b017294490e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:21.680251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2953560a-276f-4451-9ea4-3b017294490e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:21.697270 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 290/577] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:21.698626 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2953560a-276f-4451-9ea4-3b017294490e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:21.750101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:21.753403 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff4a037-d194-427d-a025-abfd90cfacdf None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:21.768591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:21.800619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66dcc8b3-bead-4306-bb16-29c6145a069f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:21.801261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66dcc8b3-bead-4306-bb16-29c6145a069f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:21.801534 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66dcc8b3-bead-4306-bb16-29c6145a069f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:21.801801 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66dcc8b3-bead-4306-bb16-29c6145a069f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:21.814287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66dcc8b3-bead-4306-bb16-29c6145a069f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:21.815756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66dcc8b3-bead-4306-bb16-29c6145a069f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:21.832692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-66dcc8b3-bead-4306-bb16-29c6145a069f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:21.832692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:21.832692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:21.832692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:21.832692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:21.832692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:21.832692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:21.832692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:21.832692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:21.832692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:21.832692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:21.832692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:21.832692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:21.832692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:21.832692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:21.832692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:21.834608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:21.834608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:21.834608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:21.834608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:21.834608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:21.834608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:21.834608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:21.834608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:21.834608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:21.834608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:21.834608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:21.834608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:21.834608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:21.834608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:21.834608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:21.834608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:21.834608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:21.836506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:21.836506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:21.836506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:21.836506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 288/578] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:21.836506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66dcc8b3-bead-4306-bb16-29c6145a069f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:21.849734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2953560a-276f-4451-9ea4-3b017294490e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:21.899803 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030bbc87-b2a5-4b75-86ab-d162368f3021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:21.900471 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030bbc87-b2a5-4b75-86ab-d162368f3021 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:21.900784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030bbc87-b2a5-4b75-86ab-d162368f3021 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:21.901061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030bbc87-b2a5-4b75-86ab-d162368f3021 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:21.915101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030bbc87-b2a5-4b75-86ab-d162368f3021 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:21.917563 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030bbc87-b2a5-4b75-86ab-d162368f3021 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:21.936205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-030bbc87-b2a5-4b75-86ab-d162368f3021 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:21.936205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:21.936205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:21.936205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:21.936205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:21.936205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:21.936205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:21.936205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:21.936205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:21.936205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:21.936205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:21.936205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:21.936205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:21.936205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:21.936205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:21.936205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:21.937316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:21.937316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:21.937316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:21.937316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:21.937316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:21.937316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:21.937316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:21.937316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:21.937316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:21.937316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:21.937316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:21.937316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:21.937316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:21.937316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:21.937316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:21.937316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:21.937316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:21.938377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:21.938377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:21.938377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:21.939417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 291/579] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:21.940753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030bbc87-b2a5-4b75-86ab-d162368f3021 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:21.966662 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66dcc8b3-bead-4306-bb16-29c6145a069f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:22.008281 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c44fab3-0dff-4a51-b03d-d1e1edfc408b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:22.008949 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c44fab3-0dff-4a51-b03d-d1e1edfc408b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:22.009231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c44fab3-0dff-4a51-b03d-d1e1edfc408b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:22.009496 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c44fab3-0dff-4a51-b03d-d1e1edfc408b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:22.018665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c44fab3-0dff-4a51-b03d-d1e1edfc408b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:22.020143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c44fab3-0dff-4a51-b03d-d1e1edfc408b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:22.033976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-3c44fab3-0dff-4a51-b03d-d1e1edfc408b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:22.033976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:22.033976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:22.033976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:22.033976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:22.033976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:22.033976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:22.033976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:22.033976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:22.033976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:22.033976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:22.033976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:22.033976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:22.033976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:22.033976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:22.035506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:22.035506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:22.035506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:22.035506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:22.035506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:22.035506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:22.035506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:22.035506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:22.035506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:22.035506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:22.035506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:22.035506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:22.035506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:22.035506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:22.035506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:22.035506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:22.035506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:22.036531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:22.036531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:22.036531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:22.036531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:22.036531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:22.036531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:22.036531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:22.036531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:22.036531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 289/580] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:22.036531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c44fab3-0dff-4a51-b03d-d1e1edfc408b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:22.050728 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:22.065010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:22.070868 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030bbc87-b2a5-4b75-86ab-d162368f3021 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:22.115398 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fce018-1b33-4a23-92e0-e8cb094506fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:22.116019 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fce018-1b33-4a23-92e0-e8cb094506fc None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:22.116335 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fce018-1b33-4a23-92e0-e8cb094506fc None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:22.116616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fce018-1b33-4a23-92e0-e8cb094506fc None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:22.117368 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fce018-1b33-4a23-92e0-e8cb094506fc None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:22.117687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fce018-1b33-4a23-92e0-e8cb094506fc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:22.119149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fce018-1b33-4a23-92e0-e8cb094506fc None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:22.139703 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 292/581] 10.222.0.64 () {56 vars in 1137 bytes} [Tue Oct 4 14:15:21 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:22.140834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fce018-1b33-4a23-92e0-e8cb094506fc None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:22.164251 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c44fab3-0dff-4a51-b03d-d1e1edfc408b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:22.202687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed69c1eb-f08a-4266-8f59-2722d94aeac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:22.203883 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed69c1eb-f08a-4266-8f59-2722d94aeac4 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:22.204259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed69c1eb-f08a-4266-8f59-2722d94aeac4 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:22.204776 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed69c1eb-f08a-4266-8f59-2722d94aeac4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:22.213910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed69c1eb-f08a-4266-8f59-2722d94aeac4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:22.215375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed69c1eb-f08a-4266-8f59-2722d94aeac4 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:22.228395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-ed69c1eb-f08a-4266-8f59-2722d94aeac4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:15:22.228395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:22.228395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:22.228395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:22.228395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:22.228395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:22.228395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:22.228395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:22.228395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:22.228395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:22.228395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:22.228395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:22.228395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:22.228395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:22.228395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:22.228395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:22.230178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:22.232000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:22.232000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:22.232000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:22.232000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:22.232000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:22.232000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:22.232000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:22.232000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:22.232000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:15:22.232000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:22.232000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 290/582] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Oct 4 14:15:22 2022] GET /identity/v3/users/nova => generated 82 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:22.232000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed69c1eb-f08a-4266-8f59-2722d94aeac4 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:22.263635 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fce018-1b33-4a23-92e0-e8cb094506fc None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:22.304957 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518e045c-0d96-4961-bda4-17b3ca6a4951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:22.305570 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518e045c-0d96-4961-bda4-17b3ca6a4951 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:22.305856 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518e045c-0d96-4961-bda4-17b3ca6a4951 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:22.306129 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518e045c-0d96-4961-bda4-17b3ca6a4951 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:22.306783 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518e045c-0d96-4961-bda4-17b3ca6a4951 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:22.307099 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518e045c-0d96-4961-bda4-17b3ca6a4951 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:22.308547 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518e045c-0d96-4961-bda4-17b3ca6a4951 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:22.324211 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 293/583] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:22.325188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518e045c-0d96-4961-bda4-17b3ca6a4951 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:22.328046 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:22.340003 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:22.358903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed69c1eb-f08a-4266-8f59-2722d94aeac4 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:22.398986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b403ebba-5983-4192-b8e5-a90a92711b78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:22.399590 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b403ebba-5983-4192-b8e5-a90a92711b78 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:22.399871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b403ebba-5983-4192-b8e5-a90a92711b78 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:22.400145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b403ebba-5983-4192-b8e5-a90a92711b78 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:22.400836 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b403ebba-5983-4192-b8e5-a90a92711b78 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:22.401142 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b403ebba-5983-4192-b8e5-a90a92711b78 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:22.402536 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b403ebba-5983-4192-b8e5-a90a92711b78 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:22.417338 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 291/584] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:22.418223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b403ebba-5983-4192-b8e5-a90a92711b78 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:22.450675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518e045c-0d96-4961-bda4-17b3ca6a4951 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:22.492281 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0f0ce0-2e9c-456a-a0a2-5275a8142f05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:22.492828 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0f0ce0-2e9c-456a-a0a2-5275a8142f05 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:22.493117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0f0ce0-2e9c-456a-a0a2-5275a8142f05 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:22.493398 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0f0ce0-2e9c-456a-a0a2-5275a8142f05 None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:22.494107 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0f0ce0-2e9c-456a-a0a2-5275a8142f05 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:22.494419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0f0ce0-2e9c-456a-a0a2-5275a8142f05 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:22.495911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0f0ce0-2e9c-456a-a0a2-5275a8142f05 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:22.509895 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 294/585] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:22.510775 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0f0ce0-2e9c-456a-a0a2-5275a8142f05 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:22.546771 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b403ebba-5983-4192-b8e5-a90a92711b78 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:22.586314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:22.590163 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fccae3-7944-4785-91c1-5f2291e4bc14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:22.590812 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fccae3-7944-4785-91c1-5f2291e4bc14 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:22.591088 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fccae3-7944-4785-91c1-5f2291e4bc14 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:22.591360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fccae3-7944-4785-91c1-5f2291e4bc14 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:22.603526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fccae3-7944-4785-91c1-5f2291e4bc14 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:22.605102 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fccae3-7944-4785-91c1-5f2291e4bc14 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:22.608498 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:22.629532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-90fccae3-7944-4785-91c1-5f2291e4bc14 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:22.629532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:22.629532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:22.629532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:22.629532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:22.629532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:22.629532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:22.629532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:22.629532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:22.629532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:22.629532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:22.629532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:22.629532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:22.629532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:22.629532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:22.629532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:22.630635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:22.630635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:22.630635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:22.630635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:22.630635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:22.630635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:22.630635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:22.630635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:22.630635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:22.630635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:22.630635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:22.630635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:22.630635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:22.630635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:22.630635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:22.630635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:22.630635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:22.631531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:22.631531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:22.631531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:22.631531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 292/586] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:22.632288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fccae3-7944-4785-91c1-5f2291e4bc14 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:22.637884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0f0ce0-2e9c-456a-a0a2-5275a8142f05 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:22.684627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3424f964-29cc-4dc8-9178-654d2c7c34db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:22.685592 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3424f964-29cc-4dc8-9178-654d2c7c34db None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:22.686038 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3424f964-29cc-4dc8-9178-654d2c7c34db None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:22.686448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3424f964-29cc-4dc8-9178-654d2c7c34db None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:22.687266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3424f964-29cc-4dc8-9178-654d2c7c34db None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:22.688245 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3424f964-29cc-4dc8-9178-654d2c7c34db None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:22.690257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3424f964-29cc-4dc8-9178-654d2c7c34db None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:22.712113 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 295/587] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Oct 4 14:15:22 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:22.713020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3424f964-29cc-4dc8-9178-654d2c7c34db None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:22.762548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fccae3-7944-4785-91c1-5f2291e4bc14 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:22.802310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3040ee65-7eda-43a5-badf-73da5df8c10c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:22.802959 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3040ee65-7eda-43a5-badf-73da5df8c10c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:22.803236 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3040ee65-7eda-43a5-badf-73da5df8c10c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:22.803506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3040ee65-7eda-43a5-badf-73da5df8c10c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:22.811785 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3040ee65-7eda-43a5-badf-73da5df8c10c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:22.814383 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3040ee65-7eda-43a5-badf-73da5df8c10c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:22.827368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-3040ee65-7eda-43a5-badf-73da5df8c10c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:22.827368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:22.827368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:22.827368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:22.827368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:22.827368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:22.827368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:22.827368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:22.827368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:22.827368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:22.827368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:22.827368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:22.827368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:22.827368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:22.827368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:22.828547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:22.828547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:22.828547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:22.828547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:22.828547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:22.828547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:22.828547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:22.828547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:22.828547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:22.828547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:22.828547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:22.828547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:22.828547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:22.828547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:22.828547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:22.828547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:22.828547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:22.830245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:22.830245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:22.830245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:22.830245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:22.830245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:22.830245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:22.830245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:22.830245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:22.831104 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3424f964-29cc-4dc8-9178-654d2c7c34db None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:22.831260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 293/588] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:22.831260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3040ee65-7eda-43a5-badf-73da5df8c10c None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:22.858575 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:22.880256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7a16af-8819-47f6-8a85-ca1d413b29e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:22.881322 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a16af-8819-47f6-8a85-ca1d413b29e4 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:22.881780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a16af-8819-47f6-8a85-ca1d413b29e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:22.882229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a16af-8819-47f6-8a85-ca1d413b29e4 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:22.892394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7a16af-8819-47f6-8a85-ca1d413b29e4 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:22.893853 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7a16af-8819-47f6-8a85-ca1d413b29e4 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:22.907157 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:22.907835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-cf7a16af-8819-47f6-8a85-ca1d413b29e4 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 14:15:22.907835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:22.907835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:22.907835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:22.907835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:22.907835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:22.907835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:22.907835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:22.907835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:22.907835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:22.907835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:22.907835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:22.907835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:22.907835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:22.907835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:22.907835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:22.909550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:22.911347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:22.911347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:22.911347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:22.911347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:22.911347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:22.911347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:22.911347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:22.911347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:22.911347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 14:15:22.911347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:22.911347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 296/589] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:22 2022] GET /identity/v3/users/neutron => generated 85 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:22.911347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7a16af-8819-47f6-8a85-ca1d413b29e4 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:22.964166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3040ee65-7eda-43a5-badf-73da5df8c10c None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:23.004208 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89d6749-e626-4f08-8af2-6f8bb8dec06d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:23.005176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89d6749-e626-4f08-8af2-6f8bb8dec06d None admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:23.005447 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89d6749-e626-4f08-8af2-6f8bb8dec06d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:23.005723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89d6749-e626-4f08-8af2-6f8bb8dec06d None admin] PATH_INFO: `/v3/projects/895ece8b70ef44adadc3416f81a80731/users/35a5a4ef40204e158649ca25583bbde7/roles/3ebe3c0be8c34c5eae533311e9b44176` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:23.022941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89d6749-e626-4f08-8af2-6f8bb8dec06d None admin] RBAC: Authorizing `identity:create_grant(project_id=895ece8b70ef44adadc3416f81a80731, user_id=35a5a4ef40204e158649ca25583bbde7, role_id=3ebe3c0be8c34c5eae533311e9b44176)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:23.024366 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89d6749-e626-4f08-8af2-6f8bb8dec06d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:23.028656 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7a16af-8819-47f6-8a85-ca1d413b29e4 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:23.040873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 294/590] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:15:22 2022] PUT /identity/v3/projects/895ece8b70ef44adadc3416f81a80731/users/35a5a4ef40204e158649ca25583bbde7/roles/3ebe3c0be8c34c5eae533311e9b44176 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:15:23.041741 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89d6749-e626-4f08-8af2-6f8bb8dec06d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:23.097668 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0d37f7-b448-4e93-8008-226fe429aeec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:23.098668 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0d37f7-b448-4e93-8008-226fe429aeec None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:23.099113 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0d37f7-b448-4e93-8008-226fe429aeec None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:23.099553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0d37f7-b448-4e93-8008-226fe429aeec None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:23.100709 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0d37f7-b448-4e93-8008-226fe429aeec None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:23.101131 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0d37f7-b448-4e93-8008-226fe429aeec None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:23.103254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0d37f7-b448-4e93-8008-226fe429aeec None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:23.126342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 297/591] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:23.126645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0d37f7-b448-4e93-8008-226fe429aeec None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:23.199936 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:23.236714 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89d6749-e626-4f08-8af2-6f8bb8dec06d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:23.248011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:23.310883 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dd721a-4fc9-4c79-bc58-bb256d2cbb65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:23.311709 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dd721a-4fc9-4c79-bc58-bb256d2cbb65 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:23.312119 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dd721a-4fc9-4c79-bc58-bb256d2cbb65 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:23.313221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dd721a-4fc9-4c79-bc58-bb256d2cbb65 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:23.322669 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0d37f7-b448-4e93-8008-226fe429aeec None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:23.331538 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07dd721a-4fc9-4c79-bc58-bb256d2cbb65 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:23.334608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07dd721a-4fc9-4c79-bc58-bb256d2cbb65 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:23.361788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-07dd721a-4fc9-4c79-bc58-bb256d2cbb65 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:23.361788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:23.361788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:23.361788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:23.361788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:23.361788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:23.361788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:23.361788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:23.361788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:23.361788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:23.361788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:23.361788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:23.361788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:23.361788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:23.361788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:23.361788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:23.363736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:23.363736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:23.363736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:23.363736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:23.363736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:23.363736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:23.363736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:23.363736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:23.363736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:23.363736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:23.363736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:23.363736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:23.363736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:23.363736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:23.363736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:23.363736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:23.363736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:23.366245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:23.366245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:23.366245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:23.366245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 295/592] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 322 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:23.370317 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dd721a-4fc9-4c79-bc58-bb256d2cbb65 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:23.396594 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee403149-0762-4fd0-aacd-d2b65aef582c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:23.398062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee403149-0762-4fd0-aacd-d2b65aef582c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:23.398510 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee403149-0762-4fd0-aacd-d2b65aef582c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:23.398947 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee403149-0762-4fd0-aacd-d2b65aef582c None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:23.400054 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee403149-0762-4fd0-aacd-d2b65aef582c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:23.400585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee403149-0762-4fd0-aacd-d2b65aef582c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:23.402633 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee403149-0762-4fd0-aacd-d2b65aef582c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:23.426100 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 298/593] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:23.428057 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee403149-0762-4fd0-aacd-d2b65aef582c None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:23.532986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dd721a-4fc9-4c79-bc58-bb256d2cbb65 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:23.597531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128239c1-bf8d-48cc-8e37-7d05903905db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:23.598618 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128239c1-bf8d-48cc-8e37-7d05903905db None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:23.599109 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128239c1-bf8d-48cc-8e37-7d05903905db None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:23.599598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128239c1-bf8d-48cc-8e37-7d05903905db None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:23.601257 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128239c1-bf8d-48cc-8e37-7d05903905db None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:23.601899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128239c1-bf8d-48cc-8e37-7d05903905db None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:23.605262 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128239c1-bf8d-48cc-8e37-7d05903905db None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:23.619946 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:23.627030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee403149-0762-4fd0-aacd-d2b65aef582c None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:23.648130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 296/594] 10.222.0.64 () {56 vars in 1139 bytes} [Tue Oct 4 14:15:23 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:23.648130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128239c1-bf8d-48cc-8e37-7d05903905db None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:23.678856 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:23.692916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6e124d-49e3-4ead-8a04-8c4acd91944e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:23.693709 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6e124d-49e3-4ead-8a04-8c4acd91944e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:23.694047 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6e124d-49e3-4ead-8a04-8c4acd91944e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:23.694469 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6e124d-49e3-4ead-8a04-8c4acd91944e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:23.704144 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6e124d-49e3-4ead-8a04-8c4acd91944e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:23.705803 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6e124d-49e3-4ead-8a04-8c4acd91944e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:23.720213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-5a6e124d-49e3-4ead-8a04-8c4acd91944e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:23.720213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:23.720213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:23.720213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:23.720213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:23.720213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:23.720213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:23.720213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:23.720213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:23.720213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:23.720213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:23.720213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:23.720213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:23.720213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:23.720213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:23.722229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:23.722229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:23.722229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:23.722229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:23.722229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:23.722229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:23.722229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:23.722229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:23.722229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:23.722229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:23.722229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:23.722229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:23.722229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:23.722229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:23.722229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:23.722229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:23.722229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:23.723833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:23.723833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:23.723833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:23.723833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:23.723833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:23.723833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:23.723833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:23.723833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:23.723833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 299/595] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:23.723833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6e124d-49e3-4ead-8a04-8c4acd91944e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:23.855870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128239c1-bf8d-48cc-8e37-7d05903905db None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:23.903584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b5d1df-9ed4-4e08-85c8-2ebe474d1e27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:23.904425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b5d1df-9ed4-4e08-85c8-2ebe474d1e27 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:23.904808 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b5d1df-9ed4-4e08-85c8-2ebe474d1e27 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:23.905447 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b5d1df-9ed4-4e08-85c8-2ebe474d1e27 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:23.906763 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b5d1df-9ed4-4e08-85c8-2ebe474d1e27 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:23.907183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b5d1df-9ed4-4e08-85c8-2ebe474d1e27 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:23.909165 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b5d1df-9ed4-4e08-85c8-2ebe474d1e27 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:23.924623 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6e124d-49e3-4ead-8a04-8c4acd91944e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:23.928252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 297/596] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:23.929888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b5d1df-9ed4-4e08-85c8-2ebe474d1e27 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:23.976826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801d7294-eec0-4287-a797-3e5a8d40312b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:23.977739 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801d7294-eec0-4287-a797-3e5a8d40312b None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:23.978016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801d7294-eec0-4287-a797-3e5a8d40312b None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:23.978321 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801d7294-eec0-4287-a797-3e5a8d40312b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:23.979062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801d7294-eec0-4287-a797-3e5a8d40312b None admin] RBAC: Adding query filter params (role.id=3ebe3c0be8c34c5eae533311e9b44176, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=5980e353164c4873bcdd2af847e0a1f1) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:23.979510 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801d7294-eec0-4287-a797-3e5a8d40312b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:23.980950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801d7294-eec0-4287-a797-3e5a8d40312b None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:23.995486 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 300/597] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:15:23 2022] GET /identity/v3/role_assignments?user.id=5980e353164c4873bcdd2af847e0a1f1&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=3ebe3c0be8c34c5eae533311e9b44176 => generated 259 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:23.995486 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801d7294-eec0-4287-a797-3e5a8d40312b None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:24.049001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:24.087015 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:24.121842 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b5d1df-9ed4-4e08-85c8-2ebe474d1e27 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:24.143089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801d7294-eec0-4287-a797-3e5a8d40312b None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:24.165296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b950eac5-af23-4ee5-891d-df15e5208181 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:24.166207 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b950eac5-af23-4ee5-891d-df15e5208181 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:24.166207 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b950eac5-af23-4ee5-891d-df15e5208181 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:24.166474 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b950eac5-af23-4ee5-891d-df15e5208181 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:24.179412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b950eac5-af23-4ee5-891d-df15e5208181 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:24.180995 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b950eac5-af23-4ee5-891d-df15e5208181 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:24.197200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a257ef-c3fd-40dd-be6c-817226bc17d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:24.198287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a257ef-c3fd-40dd-be6c-817226bc17d2 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:24.198761 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a257ef-c3fd-40dd-be6c-817226bc17d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:24.199237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a257ef-c3fd-40dd-be6c-817226bc17d2 None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:24.199642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-b950eac5-af23-4ee5-891d-df15e5208181 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:24.199642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:24.199642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:24.199642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:24.199642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:24.199642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:24.199642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:24.199642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:24.199642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:24.199642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:24.199642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:24.199642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:24.199642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:24.199642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:24.199642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:24.199642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:24.200615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:24.200615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:24.200615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:24.200615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:24.200615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:24.200615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:24.200615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:24.200615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:24.200615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:24.200615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:24.200615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:24.200615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:24.200615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:24.200615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:24.200615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:24.200615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:24.200615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:24.201517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:24.201517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:24.201517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:24.202679 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a257ef-c3fd-40dd-be6c-817226bc17d2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:24.203737 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a257ef-c3fd-40dd-be6c-817226bc17d2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:24.205495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a257ef-c3fd-40dd-be6c-817226bc17d2 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:24.206276 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 298/598] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:24.207256 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b950eac5-af23-4ee5-891d-df15e5208181 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:24.225344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 301/599] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:24.226554 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a257ef-c3fd-40dd-be6c-817226bc17d2 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:24.353875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b950eac5-af23-4ee5-891d-df15e5208181 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:24.373036 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:24.402498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a257ef-c3fd-40dd-be6c-817226bc17d2 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:24.407140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:24.425614 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b91d99-3261-44c9-a979-0a84656a6d03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:24.426647 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b91d99-3261-44c9-a979-0a84656a6d03 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:24.427126 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b91d99-3261-44c9-a979-0a84656a6d03 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:24.427799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b91d99-3261-44c9-a979-0a84656a6d03 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:24.436794 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b91d99-3261-44c9-a979-0a84656a6d03 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:24.439254 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b91d99-3261-44c9-a979-0a84656a6d03 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:24.452929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-65b91d99-3261-44c9-a979-0a84656a6d03 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:24.452929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:24.452929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:24.452929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:24.452929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:24.452929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:24.452929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:24.452929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:24.452929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:24.452929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:24.452929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:24.452929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:24.452929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:24.452929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:24.452929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:24.454453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:24.454453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:24.454453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:24.454453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:24.454453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:24.454453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:24.454453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:24.454453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:24.454453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:24.454453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:24.454453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:24.454453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:24.454453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:24.454453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:24.454453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:24.454453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:24.454453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:24.455411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:24.455411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:24.455411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:24.455411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:24.455411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:24.455411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:24.455411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:24.455411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:24.458420 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 299/600] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:24.459492 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b91d99-3261-44c9-a979-0a84656a6d03 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:24.475346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c49f74c-2583-462f-a306-1dbe62866d25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:24.476650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c49f74c-2583-462f-a306-1dbe62866d25 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:24.477224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c49f74c-2583-462f-a306-1dbe62866d25 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:24.477863 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c49f74c-2583-462f-a306-1dbe62866d25 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:24.479131 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c49f74c-2583-462f-a306-1dbe62866d25 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:24.480078 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c49f74c-2583-462f-a306-1dbe62866d25 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:24.482603 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c49f74c-2583-462f-a306-1dbe62866d25 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:24.511396 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 302/601] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:24.511396 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c49f74c-2583-462f-a306-1dbe62866d25 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:24.648089 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b91d99-3261-44c9-a979-0a84656a6d03 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:24.700359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f161c2-830c-44d8-9180-f55f727848c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:24.701370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f161c2-830c-44d8-9180-f55f727848c3 None admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:24.701370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f161c2-830c-44d8-9180-f55f727848c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:24.701674 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f161c2-830c-44d8-9180-f55f727848c3 None admin] PATH_INFO: `/v3/projects/895ece8b70ef44adadc3416f81a80731/users/058c027ade94498fbab7b9347e87c9c3/roles/3ebe3c0be8c34c5eae533311e9b44176` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:24.721617 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f161c2-830c-44d8-9180-f55f727848c3 None admin] RBAC: Authorizing `identity:create_grant(project_id=895ece8b70ef44adadc3416f81a80731, user_id=058c027ade94498fbab7b9347e87c9c3, role_id=3ebe3c0be8c34c5eae533311e9b44176)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:24.723116 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f161c2-830c-44d8-9180-f55f727848c3 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:24.723800 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c49f74c-2583-462f-a306-1dbe62866d25 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:24.739538 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 300/602] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:15:24 2022] PUT /identity/v3/projects/895ece8b70ef44adadc3416f81a80731/users/058c027ade94498fbab7b9347e87c9c3/roles/3ebe3c0be8c34c5eae533311e9b44176 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:15:24.740855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f161c2-830c-44d8-9180-f55f727848c3 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:24.792442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682794b0-c546-40ae-b7d4-9265143e0949 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:24.793488 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:24.795887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682794b0-c546-40ae-b7d4-9265143e0949 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:24.797356 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682794b0-c546-40ae-b7d4-9265143e0949 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:24.798330 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682794b0-c546-40ae-b7d4-9265143e0949 None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:24.800863 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682794b0-c546-40ae-b7d4-9265143e0949 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:24.801786 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682794b0-c546-40ae-b7d4-9265143e0949 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:24.805870 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682794b0-c546-40ae-b7d4-9265143e0949 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:24.833883 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 303/603] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:24.850418 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682794b0-c546-40ae-b7d4-9265143e0949 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:24.858921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:24.879814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f161c2-830c-44d8-9180-f55f727848c3 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:24.924817 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3dcef6-9257-4106-a0b7-1135cdfe278f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:24.925846 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3dcef6-9257-4106-a0b7-1135cdfe278f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:24.926337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3dcef6-9257-4106-a0b7-1135cdfe278f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:24.926817 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3dcef6-9257-4106-a0b7-1135cdfe278f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:24.935456 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3dcef6-9257-4106-a0b7-1135cdfe278f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:24.937577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3dcef6-9257-4106-a0b7-1135cdfe278f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:24.951626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-4d3dcef6-9257-4106-a0b7-1135cdfe278f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:24.951626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:24.951626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:24.951626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:24.951626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:24.951626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:24.951626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:24.951626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:24.951626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:24.951626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:24.951626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:24.951626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:24.951626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:24.951626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:24.951626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:24.952682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:24.952682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:24.952682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:24.952682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:24.952682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:24.952682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:24.952682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:24.952682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:24.952682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:24.952682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:24.952682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:24.952682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:24.952682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:24.952682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:24.952682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:24.952682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:24.952682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:24.954482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:24.954482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:24.954482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:24.954482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:24.954482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:24.954482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:24.954482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:24.954482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:24.957486 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 301/604] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:24.967650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3dcef6-9257-4106-a0b7-1135cdfe278f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:25.139845 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682794b0-c546-40ae-b7d4-9265143e0949 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:25.217422 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad5501e-8507-4d71-b8c0-fa6b09236559 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:25.218483 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad5501e-8507-4d71-b8c0-fa6b09236559 None admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:25.219237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad5501e-8507-4d71-b8c0-fa6b09236559 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:25.219417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad5501e-8507-4d71-b8c0-fa6b09236559 None admin] PATH_INFO: `/v3/projects/895ece8b70ef44adadc3416f81a80731/users/03081d7d581745c88c12e9d089fcb2d3/roles/3ebe3c0be8c34c5eae533311e9b44176` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:25.225352 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3dcef6-9257-4106-a0b7-1135cdfe278f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:25.228109 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:25.269942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad5501e-8507-4d71-b8c0-fa6b09236559 None admin] RBAC: Authorizing `identity:create_grant(project_id=895ece8b70ef44adadc3416f81a80731, user_id=03081d7d581745c88c12e9d089fcb2d3, role_id=3ebe3c0be8c34c5eae533311e9b44176)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:25.272696 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad5501e-8507-4d71-b8c0-fa6b09236559 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:25.284029 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d43a49f-0c2d-46a2-8da2-196317104da9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:25.284762 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d43a49f-0c2d-46a2-8da2-196317104da9 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:25.285122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d43a49f-0c2d-46a2-8da2-196317104da9 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:25.285656 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d43a49f-0c2d-46a2-8da2-196317104da9 None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:25.286491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d43a49f-0c2d-46a2-8da2-196317104da9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:25.286794 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d43a49f-0c2d-46a2-8da2-196317104da9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:25.288344 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d43a49f-0c2d-46a2-8da2-196317104da9 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:25.305111 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 302/605] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 339 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:15:25.316007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d43a49f-0c2d-46a2-8da2-196317104da9 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:25.319571 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 304/606] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:15:24 2022] PUT /identity/v3/projects/895ece8b70ef44adadc3416f81a80731/users/03081d7d581745c88c12e9d089fcb2d3/roles/3ebe3c0be8c34c5eae533311e9b44176 => generated 0 bytes in 469 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:15:25.449439 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d43a49f-0c2d-46a2-8da2-196317104da9 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:25.493434 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2371f5-c21b-4649-a97a-352ef2daa6c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:25.494011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2371f5-c21b-4649-a97a-352ef2daa6c0 None admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:25.494289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2371f5-c21b-4649-a97a-352ef2daa6c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:25.494565 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2371f5-c21b-4649-a97a-352ef2daa6c0 None admin] PATH_INFO: `/v3/projects/895ece8b70ef44adadc3416f81a80731/users/8dbeccfb9a194992959f0b341ef0d4a5/roles/3ebe3c0be8c34c5eae533311e9b44176` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:25.512796 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2371f5-c21b-4649-a97a-352ef2daa6c0 None admin] RBAC: Authorizing `identity:create_grant(project_id=895ece8b70ef44adadc3416f81a80731, user_id=8dbeccfb9a194992959f0b341ef0d4a5, role_id=3ebe3c0be8c34c5eae533311e9b44176)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:25.514398 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2371f5-c21b-4649-a97a-352ef2daa6c0 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:25.539321 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 303/607] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:15:25 2022] PUT /identity/v3/projects/895ece8b70ef44adadc3416f81a80731/users/8dbeccfb9a194992959f0b341ef0d4a5/roles/3ebe3c0be8c34c5eae533311e9b44176 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:15:26.295984 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8de68c-f2a3-4bbd-ab28-25302d5b3310 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:26.297367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8de68c-f2a3-4bbd-ab28-25302d5b3310 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:26.297506 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8de68c-f2a3-4bbd-ab28-25302d5b3310 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:26.300372 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 305/608] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:26 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:26.306928 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7124f5f9-b985-49dd-884e-330d8dc8015c None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:26.307715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7124f5f9-b985-49dd-884e-330d8dc8015c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:26.307715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7124f5f9-b985-49dd-884e-330d8dc8015c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:26.348649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-7124f5f9-b985-49dd-884e-330d8dc8015c None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:26.350423 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7124f5f9-b985-49dd-884e-330d8dc8015c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:26.354045 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:26.526812 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 304/609] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:26.927721 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8de68c-f2a3-4bbd-ab28-25302d5b3310 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:27.080732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:27.211634 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8de68c-f2a3-4bbd-ab28-25302d5b3310 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:27.290156 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7946fe9f-ebe1-4b03-9891-ca9743cda3b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:27.291218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7946fe9f-ebe1-4b03-9891-ca9743cda3b5 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:27.291701 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7946fe9f-ebe1-4b03-9891-ca9743cda3b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:27.292178 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7946fe9f-ebe1-4b03-9891-ca9743cda3b5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:27.321933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7946fe9f-ebe1-4b03-9891-ca9743cda3b5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:27.332818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-7946fe9f-ebe1-4b03-9891-ca9743cda3b5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:27.332818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:27.332818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:27.332818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:27.332818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:27.332818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:27.332818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:27.332818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:27.332818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:27.332818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:27.332818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:27.332818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:27.332818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:27.332818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:27.332818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:27.332818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:27.334599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:27.334599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:27.334599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:27.334599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:27.334599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:27.334599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:27.334599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:27.334599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:27.334599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:27.334599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:27.334599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:27.334599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:27.334599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:27.334599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:27.334599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:27.334599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:27.336185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:27.336185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:27.336185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:27.336185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:27.336185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:27.336185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:27.336185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:27.336185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:27.336185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:27.336185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:27.336185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:27.336185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:27.345053 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 306/610] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 418 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:27.353506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7124f5f9-b985-49dd-884e-330d8dc8015c None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:27.498182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7124f5f9-b985-49dd-884e-330d8dc8015c None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:27.553587 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755731c9-e706-4d20-b085-1159d96b909f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:27.554230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755731c9-e706-4d20-b085-1159d96b909f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:27.554510 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755731c9-e706-4d20-b085-1159d96b909f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:27.554798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755731c9-e706-4d20-b085-1159d96b909f None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:27.555482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755731c9-e706-4d20-b085-1159d96b909f None admin] RBAC: Adding query filter params (name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:27.555787 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755731c9-e706-4d20-b085-1159d96b909f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:27.557316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755731c9-e706-4d20-b085-1159d96b909f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:27.564303 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:27.572394 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 305/611] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:15:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:27.579395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7946fe9f-ebe1-4b03-9891-ca9743cda3b5 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:27.792683 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7946fe9f-ebe1-4b03-9891-ca9743cda3b5 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:27.822818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:27.838014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9488f41e-f4f3-4323-9e2d-483535527c5f None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:27.838692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9488f41e-f4f3-4323-9e2d-483535527c5f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:27.839189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9488f41e-f4f3-4323-9e2d-483535527c5f None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:27.841127 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 306/612] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:27.849454 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9bcccc-f532-4aee-8490-104857710110 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:27.849854 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9bcccc-f532-4aee-8490-104857710110 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:27.850312 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9bcccc-f532-4aee-8490-104857710110 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:27.862055 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3650162c-b270-423d-aecb-764b8a59d7ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:27.863229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3650162c-b270-423d-aecb-764b8a59d7ce None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:27.863721 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3650162c-b270-423d-aecb-764b8a59d7ce None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:27.864218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3650162c-b270-423d-aecb-764b8a59d7ce None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:27.881700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-ac9bcccc-f532-4aee-8490-104857710110 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:27.883778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ac9bcccc-f532-4aee-8490-104857710110 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:27.887034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3650162c-b270-423d-aecb-764b8a59d7ce None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:27.889070 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3650162c-b270-423d-aecb-764b8a59d7ce None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:27.919373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-3650162c-b270-423d-aecb-764b8a59d7ce None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:27.919373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:27.919373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:27.919373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:27.919373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:27.919373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:27.919373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:27.919373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:27.919373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:27.919373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:27.919373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:27.919373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:27.919373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:27.919373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:27.919373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:27.919373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:27.920454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:27.920454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:27.920454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:27.920454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:27.920454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:27.920454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:27.920454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:27.920454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:27.920454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:27.920454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:27.920454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:27.920454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:27.920454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:27.920454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:27.920454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:27.920454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:27.920454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:27.921352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:27.921352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:27.921352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:27.923882 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 307/613] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 346 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:27.933648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3650162c-b270-423d-aecb-764b8a59d7ce None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:28.020158 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 307/614] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:28.247220 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3650162c-b270-423d-aecb-764b8a59d7ce None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:28.347681 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b8224b-2528-4d95-b80d-a7b2b01ab8e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:28.348668 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b8224b-2528-4d95-b80d-a7b2b01ab8e8 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:28.349090 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b8224b-2528-4d95-b80d-a7b2b01ab8e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:28.349517 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b8224b-2528-4d95-b80d-a7b2b01ab8e8 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:28.350692 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b8224b-2528-4d95-b80d-a7b2b01ab8e8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:28.351160 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b8224b-2528-4d95-b80d-a7b2b01ab8e8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:28.353627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b8224b-2528-4d95-b80d-a7b2b01ab8e8 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:28.354917 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9bcccc-f532-4aee-8490-104857710110 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:28.396835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 308/615] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 464 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:15:28.407111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b8224b-2528-4d95-b80d-a7b2b01ab8e8 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:28.506080 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:28.676346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b8224b-2528-4d95-b80d-a7b2b01ab8e8 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:28.676717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9bcccc-f532-4aee-8490-104857710110 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:28.731929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:28.741766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4658c3-c7ab-431a-99eb-a9dd14dc886d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:28.742806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4658c3-c7ab-431a-99eb-a9dd14dc886d None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:28.743466 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4658c3-c7ab-431a-99eb-a9dd14dc886d None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:28.744059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4658c3-c7ab-431a-99eb-a9dd14dc886d None admin] PATH_INFO: `/v3/users/glance` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:28.745943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcac136-484a-45a1-aca5-45e347e62d24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:28.747329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcac136-484a-45a1-aca5-45e347e62d24 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:28.748018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcac136-484a-45a1-aca5-45e347e62d24 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:28.748738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcac136-484a-45a1-aca5-45e347e62d24 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:28.758344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4658c3-c7ab-431a-99eb-a9dd14dc886d None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:28.760904 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4658c3-c7ab-431a-99eb-a9dd14dc886d None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:28.770218 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcac136-484a-45a1-aca5-45e347e62d24 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:28.779795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-dfcac136-484a-45a1-aca5-45e347e62d24 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:28.779795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:28.779795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:28.779795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:28.779795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:28.779795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:28.779795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:28.779795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:28.779795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:28.779795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:28.779795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:28.779795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:28.779795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:28.779795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:28.779795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:28.779795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:28.782619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:28.782619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:28.782619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:28.782619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:28.782619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:28.782619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:28.782619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:28.782619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:28.782619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:28.782619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:28.782619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:28.782619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:28.782619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:28.782619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:28.782619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:28.782619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:28.784584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:28.784584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:28.784584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:28.784584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:28.784584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:28.784584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:28.784584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:28.784584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:28.784584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:28.784584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:28.784584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:28.784584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:28.786002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-4a4658c3-c7ab-431a-99eb-a9dd14dc886d None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 04 14:15:28.786002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:28.786002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:28.786002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:28.786002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:28.786002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:28.786002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:28.786002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:28.786002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:28.786002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:28.786002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:28.786002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:28.786002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:28.786002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:28.786002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:28.786002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:28.788581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:28.794342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:28.794342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:28.794342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:28.794342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:28.794342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:28.794342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:28.794342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:28.794342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:28.794342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 04 14:15:28.794342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:28.794342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 309/616] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:15:28 2022] GET /identity/v3/users/glance => generated 84 bytes in 382 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:28.794342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f591f95-d699-49f4-8bbe-824e6c2c0e26 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:28.794342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f591f95-d699-49f4-8bbe-824e6c2c0e26 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:28.797780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f591f95-d699-49f4-8bbe-824e6c2c0e26 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:28.798339 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 310/618] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:28 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:28.798595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 308/618] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 444 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:28.799531 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f591f95-d699-49f4-8bbe-824e6c2c0e26 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:28.806060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057b824-34c8-4533-92a3-1966e1b92023 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:28.806967 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057b824-34c8-4533-92a3-1966e1b92023 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:28.807685 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057b824-34c8-4533-92a3-1966e1b92023 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:28.840348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-6057b824-34c8-4533-92a3-1966e1b92023 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:28.842738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6057b824-34c8-4533-92a3-1966e1b92023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:28.948263 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 309/619] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:15:28.949861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6057b824-34c8-4533-92a3-1966e1b92023 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:28.979149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:29.023665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f591f95-d699-49f4-8bbe-824e6c2c0e26 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:29.073912 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6057b824-34c8-4533-92a3-1966e1b92023 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:29.100184 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:29.119406 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a452d55d-114f-49ad-9839-06b83a112843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:29.120513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a452d55d-114f-49ad-9839-06b83a112843 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:29.121061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a452d55d-114f-49ad-9839-06b83a112843 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:29.121539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a452d55d-114f-49ad-9839-06b83a112843 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:29.122434 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a452d55d-114f-49ad-9839-06b83a112843 None admin] RBAC: Adding query filter params (name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:29.122978 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a452d55d-114f-49ad-9839-06b83a112843 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:29.125162 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a452d55d-114f-49ad-9839-06b83a112843 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:29.135964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 310/620] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:15:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:29.149155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330897a7-4175-41b3-9262-9213ecdb8444 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:29.150155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330897a7-4175-41b3-9262-9213ecdb8444 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:29.150583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330897a7-4175-41b3-9262-9213ecdb8444 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:29.151009 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330897a7-4175-41b3-9262-9213ecdb8444 None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:29.152199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330897a7-4175-41b3-9262-9213ecdb8444 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:29.155871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330897a7-4175-41b3-9262-9213ecdb8444 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:29.157014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a452d55d-114f-49ad-9839-06b83a112843 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:29.158662 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330897a7-4175-41b3-9262-9213ecdb8444 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:29.200302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 311/621] 10.222.0.64 () {56 vars in 1137 bytes} [Tue Oct 4 14:15:28 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 401 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:29.209402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330897a7-4175-41b3-9262-9213ecdb8444 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:29.293189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a452d55d-114f-49ad-9839-06b83a112843 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:29.338919 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b78ed15-66f9-4a72-8565-20c047839eba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:29.339950 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b78ed15-66f9-4a72-8565-20c047839eba None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:29.340524 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b78ed15-66f9-4a72-8565-20c047839eba None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:29.341004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b78ed15-66f9-4a72-8565-20c047839eba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:29.353824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b78ed15-66f9-4a72-8565-20c047839eba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:29.355779 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b78ed15-66f9-4a72-8565-20c047839eba None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:29.384816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-8b78ed15-66f9-4a72-8565-20c047839eba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:29.384816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:29.384816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:29.384816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:29.384816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:29.384816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:29.384816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:29.384816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:29.384816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:29.384816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:29.384816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:29.384816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:29.384816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:29.384816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:29.384816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:29.384816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:29.386108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:29.386108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:29.386108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:29.386108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:29.386108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:29.386108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:29.386108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:29.386108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:29.386108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:29.386108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:29.386108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:29.386108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:29.386108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:29.386108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:29.386108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:29.386108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:29.386108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:29.387206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:29.387206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:29.387206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:29.387206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 311/622] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:29.389266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f37ae85-672a-49b5-b96b-96e8d23cf74b None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:29.389912 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f37ae85-672a-49b5-b96b-96e8d23cf74b None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:29.391283 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f37ae85-672a-49b5-b96b-96e8d23cf74b None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:29.394380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 312/623] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:29 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:29.397218 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6bd8d3-17ed-43e0-b57d-4da97cb69a70 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:29.397487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6bd8d3-17ed-43e0-b57d-4da97cb69a70 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:29.398409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6bd8d3-17ed-43e0-b57d-4da97cb69a70 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:29.400848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 313/624] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:29 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:29.402337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6bd8d3-17ed-43e0-b57d-4da97cb69a70 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:29.428333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330897a7-4175-41b3-9262-9213ecdb8444 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:29.472817 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75811153-6f00-41b3-865b-111641f65d00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:29.473590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75811153-6f00-41b3-865b-111641f65d00 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:29.473865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75811153-6f00-41b3-865b-111641f65d00 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:29.474159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75811153-6f00-41b3-865b-111641f65d00 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:29.474330 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:29.503355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:29.585869 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75811153-6f00-41b3-865b-111641f65d00 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:29.587876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75811153-6f00-41b3-865b-111641f65d00 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:29.598985 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6bd8d3-17ed-43e0-b57d-4da97cb69a70 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:29.650241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6181b426-0706-4a7e-804c-25273bb45e76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:29.650995 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6181b426-0706-4a7e-804c-25273bb45e76 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:29.651173 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6181b426-0706-4a7e-804c-25273bb45e76 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:29.651395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6181b426-0706-4a7e-804c-25273bb45e76 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:29.652034 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6181b426-0706-4a7e-804c-25273bb45e76 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:29.652342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6181b426-0706-4a7e-804c-25273bb45e76 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:29.653737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6181b426-0706-4a7e-804c-25273bb45e76 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:29.667691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 314/625] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:29.669021 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb8cb90-c2b9-4cec-b3fa-a5082b080f48 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:29.669298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb8cb90-c2b9-4cec-b3fa-a5082b080f48 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:29.669564 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb8cb90-c2b9-4cec-b3fa-a5082b080f48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:29.689574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-3fb8cb90-c2b9-4cec-b3fa-a5082b080f48 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:29.690778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-3fb8cb90-c2b9-4cec-b3fa-a5082b080f48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:29.723358 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 312/626] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:15:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 510 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 14:15:29.723358 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3c0396-f3d3-49ea-ab82-2e5f1904ae92 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:29.723358 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3c0396-f3d3-49ea-ab82-2e5f1904ae92 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:29.723358 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3c0396-f3d3-49ea-ab82-2e5f1904ae92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:29.740024 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-6a3c0396-f3d3-49ea-ab82-2e5f1904ae92 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:29.741316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-6a3c0396-f3d3-49ea-ab82-2e5f1904ae92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:29.785096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 315/627] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:29.786882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb8cb90-c2b9-4cec-b3fa-a5082b080f48 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:29.792925 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:29.823175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:29.832247 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 313/628] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:29.839470 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3c0396-f3d3-49ea-ab82-2e5f1904ae92 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:29.959471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb8cb90-c2b9-4cec-b3fa-a5082b080f48 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:30.043186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27b316c-e3df-4760-966f-a390277d2efd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:30.045042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27b316c-e3df-4760-966f-a390277d2efd None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:30.045829 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27b316c-e3df-4760-966f-a390277d2efd None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:30.047464 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27b316c-e3df-4760-966f-a390277d2efd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:30.068044 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27b316c-e3df-4760-966f-a390277d2efd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:30.069810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3c0396-f3d3-49ea-ab82-2e5f1904ae92 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:30.078374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-b27b316c-e3df-4760-966f-a390277d2efd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:30.078374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:30.078374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:30.078374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:30.078374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:30.078374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:30.078374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:30.078374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:30.078374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:30.078374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:30.078374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:30.078374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:30.078374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:30.078374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:30.078374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:30.078374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:30.079557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:30.079557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:30.079557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:30.079557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:30.079557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:30.079557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:30.079557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:30.079557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:30.079557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:30.079557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:30.079557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:30.079557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:30.079557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:30.079557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:30.079557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:30.079557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:30.080462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:30.080462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:30.080462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:30.080462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:30.080462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:30.080462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:30.080462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:30.080462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:30.080462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:30.080462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:30.080462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:30.080462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:30.081105 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 316/629] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:29 2022] GET /identity/v3/roles/service => generated 85 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:30.082645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27b316c-e3df-4760-966f-a390277d2efd None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:30.122609 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbe61cb-03a9-4b6c-955f-389e436b5c3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:30.123498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbe61cb-03a9-4b6c-955f-389e436b5c3b None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:30.123901 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbe61cb-03a9-4b6c-955f-389e436b5c3b None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:30.124230 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbe61cb-03a9-4b6c-955f-389e436b5c3b None admin] PATH_INFO: `/v3/users/swift` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:30.134046 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbe61cb-03a9-4b6c-955f-389e436b5c3b None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:30.135603 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbe61cb-03a9-4b6c-955f-389e436b5c3b None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:30.151976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-7dbe61cb-03a9-4b6c-955f-389e436b5c3b None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 14:15:30.151976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:30.151976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:30.151976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:30.151976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:30.151976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:30.151976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:30.151976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:30.151976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:30.151976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:30.151976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:30.151976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:30.151976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:30.151976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:30.151976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:30.151976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:30.153185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:30.154149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:30.154149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:30.154149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:30.154149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:30.154149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:30.154149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:30.154149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:30.154149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:30.154149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 14:15:30.154149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:30.154149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 314/630] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:15:29 2022] GET /identity/v3/users/swift => generated 83 bytes in 321 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:30.154741 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbe61cb-03a9-4b6c-955f-389e436b5c3b None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:30.206057 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:30.222049 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:30.242435 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27b316c-e3df-4760-966f-a390277d2efd None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:30.284342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561afc6c-2f4a-484e-b386-76baad84b735 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:30.284985 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561afc6c-2f4a-484e-b386-76baad84b735 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:30.285260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561afc6c-2f4a-484e-b386-76baad84b735 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:30.285531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561afc6c-2f4a-484e-b386-76baad84b735 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:30.298348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561afc6c-2f4a-484e-b386-76baad84b735 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:30.300908 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561afc6c-2f4a-484e-b386-76baad84b735 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:30.319933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-561afc6c-2f4a-484e-b386-76baad84b735 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:30.319933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:30.319933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:30.319933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:30.319933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:30.319933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:30.319933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:30.319933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:30.319933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:30.319933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:30.319933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:30.319933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:30.319933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:30.319933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:30.319933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:30.319933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:30.321210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:30.321210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:30.321210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:30.321210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:30.321210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:30.321210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:30.321210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:30.321210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:30.321210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:30.321210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:30.321210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:30.321210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:30.321210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:30.321210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:30.321210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:30.321210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:30.321210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:30.323410 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:30.323410 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:30.323410 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:30.323410 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 317/631] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:30.323410 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561afc6c-2f4a-484e-b386-76baad84b735 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:30.323963 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbe61cb-03a9-4b6c-955f-389e436b5c3b None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:30.389984 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e86eaf-059f-4f34-8f13-fca28ca3fe1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:30.391059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e86eaf-059f-4f34-8f13-fca28ca3fe1f None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:30.391546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e86eaf-059f-4f34-8f13-fca28ca3fe1f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:30.392032 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e86eaf-059f-4f34-8f13-fca28ca3fe1f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:30.412190 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e86eaf-059f-4f34-8f13-fca28ca3fe1f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:30.423254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-04e86eaf-059f-4f34-8f13-fca28ca3fe1f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:30.423254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:30.423254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:30.423254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:30.423254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:30.423254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:30.423254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:30.423254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:30.423254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:30.423254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:30.423254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:30.423254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:30.423254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:30.423254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:30.423254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:30.423254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:30.424405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:30.424405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:30.424405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:30.424405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:30.424405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:30.424405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:30.424405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:30.424405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:30.424405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:30.424405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:30.424405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:30.424405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:30.424405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:30.424405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:30.424405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:30.424405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:30.425256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:30.425256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:30.425256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:30.425256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:30.425256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:30.425256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:30.425256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:30.425256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:30.425256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:30.425256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:30.425256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:30.425256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:30.425937 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 315/632] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:30.426988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e86eaf-059f-4f34-8f13-fca28ca3fe1f None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:30.442551 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561afc6c-2f4a-484e-b386-76baad84b735 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:30.487010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c04115c-3dbc-4b82-b193-a19c62d6c9e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:30.487929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c04115c-3dbc-4b82-b193-a19c62d6c9e4 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:30.488451 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c04115c-3dbc-4b82-b193-a19c62d6c9e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:30.488845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c04115c-3dbc-4b82-b193-a19c62d6c9e4 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:30.489638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c04115c-3dbc-4b82-b193-a19c62d6c9e4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:30.490112 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c04115c-3dbc-4b82-b193-a19c62d6c9e4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:30.492053 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c04115c-3dbc-4b82-b193-a19c62d6c9e4 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:30.495211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:30.506431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 318/633] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:15:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:30.507825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c04115c-3dbc-4b82-b193-a19c62d6c9e4 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:30.582601 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:30.593354 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e86eaf-059f-4f34-8f13-fca28ca3fe1f None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:30.646943 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8491e23c-c831-4daf-bbdd-75119ab68eac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:30.647467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8491e23c-c831-4daf-bbdd-75119ab68eac None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:30.647749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8491e23c-c831-4daf-bbdd-75119ab68eac None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:30.648025 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8491e23c-c831-4daf-bbdd-75119ab68eac None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:30.648845 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8491e23c-c831-4daf-bbdd-75119ab68eac None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:30.649152 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8491e23c-c831-4daf-bbdd-75119ab68eac None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:30.650639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8491e23c-c831-4daf-bbdd-75119ab68eac None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:30.657876 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c04115c-3dbc-4b82-b193-a19c62d6c9e4 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:30.672171 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 316/634] 10.222.0.64 () {56 vars in 1135 bytes} [Tue Oct 4 14:15:30 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:30.673216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8491e23c-c831-4daf-bbdd-75119ab68eac None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:30.704840 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a720c259-7e06-4d84-ab8e-399d8bacab27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:30.705487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a720c259-7e06-4d84-ab8e-399d8bacab27 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:30.705756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a720c259-7e06-4d84-ab8e-399d8bacab27 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:30.706018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a720c259-7e06-4d84-ab8e-399d8bacab27 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:30.718988 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a720c259-7e06-4d84-ab8e-399d8bacab27 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:30.724472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-a720c259-7e06-4d84-ab8e-399d8bacab27 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:30.724472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:30.724472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:30.724472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:30.724472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:30.724472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:30.724472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:30.724472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:30.724472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:30.724472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:30.724472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:30.724472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:30.724472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:30.724472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:30.724472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:30.724472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:30.726390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:30.726390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:30.726390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:30.726390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:30.726390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:30.726390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:30.726390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:30.726390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:30.726390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:30.726390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:30.726390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:30.726390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:30.726390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:30.726390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:30.726390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:30.726390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:30.728019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:30.728019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:30.728019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:30.728019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:30.728019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:30.728019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:30.728019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:30.728019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:30.728019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:30.728019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:30.728019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:30.728019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:30.728019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 319/635] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:30.728019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a720c259-7e06-4d84-ab8e-399d8bacab27 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:30.823491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:30.846336 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8491e23c-c831-4daf-bbdd-75119ab68eac None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:30.876290 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a720c259-7e06-4d84-ab8e-399d8bacab27 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:30.902341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aff0bf8-8074-4fc0-8469-a9b3aac6dc8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:30.903199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aff0bf8-8074-4fc0-8469-a9b3aac6dc8e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:30.903643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aff0bf8-8074-4fc0-8469-a9b3aac6dc8e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:30.904080 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aff0bf8-8074-4fc0-8469-a9b3aac6dc8e None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:30.905481 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aff0bf8-8074-4fc0-8469-a9b3aac6dc8e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:30.906708 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aff0bf8-8074-4fc0-8469-a9b3aac6dc8e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:30.909068 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aff0bf8-8074-4fc0-8469-a9b3aac6dc8e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:30.929327 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 317/636] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:30.930008 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f429c0-8120-46ec-ab27-85f1b0b4f8fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:30.930459 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aff0bf8-8074-4fc0-8469-a9b3aac6dc8e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:30.931028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f429c0-8120-46ec-ab27-85f1b0b4f8fa None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:30.931371 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f429c0-8120-46ec-ab27-85f1b0b4f8fa None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:30.931816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f429c0-8120-46ec-ab27-85f1b0b4f8fa None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:30.932962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f429c0-8120-46ec-ab27-85f1b0b4f8fa None admin] RBAC: Adding query filter params (name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:30.933435 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f429c0-8120-46ec-ab27-85f1b0b4f8fa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:30.935617 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f429c0-8120-46ec-ab27-85f1b0b4f8fa None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:30.945727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 320/637] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:15:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:30.946592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f429c0-8120-46ec-ab27-85f1b0b4f8fa None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:30.960111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:31.078214 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f429c0-8120-46ec-ab27-85f1b0b4f8fa None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:31.089528 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aff0bf8-8074-4fc0-8469-a9b3aac6dc8e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:31.102972 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:31.121320 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd7658c-bef9-4262-8b04-b5f38d3ab7bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:31.122623 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd7658c-bef9-4262-8b04-b5f38d3ab7bc None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:31.123060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd7658c-bef9-4262-8b04-b5f38d3ab7bc None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:31.123495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd7658c-bef9-4262-8b04-b5f38d3ab7bc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:31.134489 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cccf31e-acf0-4982-a733-9a9a1a4bab93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:31.135494 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cccf31e-acf0-4982-a733-9a9a1a4bab93 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:31.135949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cccf31e-acf0-4982-a733-9a9a1a4bab93 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:31.136441 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cccf31e-acf0-4982-a733-9a9a1a4bab93 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:31.143279 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd7658c-bef9-4262-8b04-b5f38d3ab7bc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:31.145881 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd7658c-bef9-4262-8b04-b5f38d3ab7bc None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:31.155497 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cccf31e-acf0-4982-a733-9a9a1a4bab93 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:31.157902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cccf31e-acf0-4982-a733-9a9a1a4bab93 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:31.167722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-6dd7658c-bef9-4262-8b04-b5f38d3ab7bc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:31.167722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:31.167722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:31.167722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:31.167722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:31.167722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:31.167722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:31.167722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:31.167722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:31.167722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:31.167722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:31.167722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:31.167722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:31.167722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:31.167722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:31.167722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:31.169717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:31.169717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:31.169717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:31.169717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:31.169717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:31.169717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:31.169717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:31.169717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:31.169717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:31.169717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:31.169717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:31.169717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:31.169717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:31.169717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:31.169717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:31.169717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:31.169717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:31.171482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:31.171482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:31.171482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:31.171735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 321/638] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:31.173609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd7658c-bef9-4262-8b04-b5f38d3ab7bc None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:31.181149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-4cccf31e-acf0-4982-a733-9a9a1a4bab93 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:31.181149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:31.181149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:31.181149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:31.181149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:31.181149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:31.181149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:31.181149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:31.181149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:31.181149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:31.181149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:31.181149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:31.181149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:31.181149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:31.181149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:31.181149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:31.183535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:31.183535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:31.183535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:31.183535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:31.183535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:31.183535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:31.183535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:31.183535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:31.183535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:31.183535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:31.183535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:31.183535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:31.183535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:31.183535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:31.183535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:31.183535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:31.183535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:31.186341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:31.186341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:31.186341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:31.186341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 318/639] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:31.186341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cccf31e-acf0-4982-a733-9a9a1a4bab93 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:31.268593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:31.314085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cccf31e-acf0-4982-a733-9a9a1a4bab93 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:31.347121 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd7658c-bef9-4262-8b04-b5f38d3ab7bc None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:31.350769 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac32f62-cacd-4f36-be05-9c44a99fd2cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:31.351419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac32f62-cacd-4f36-be05-9c44a99fd2cc None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:31.351689 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac32f62-cacd-4f36-be05-9c44a99fd2cc None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:31.351958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac32f62-cacd-4f36-be05-9c44a99fd2cc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:31.359302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac32f62-cacd-4f36-be05-9c44a99fd2cc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:31.360808 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac32f62-cacd-4f36-be05-9c44a99fd2cc None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:31.372736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-dac32f62-cacd-4f36-be05-9c44a99fd2cc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:31.372736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:31.372736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:31.372736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:31.372736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:31.372736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:31.372736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:31.372736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:31.372736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:31.372736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:31.372736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:31.372736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:31.372736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:31.372736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:31.372736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:31.374498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:31.374498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:31.374498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:31.374498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:31.374498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:31.374498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:31.374498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:31.374498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:31.374498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:31.374498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:31.374498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:31.374498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:31.374498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:31.374498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:31.374498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:31.374498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:31.374498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:31.376814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:31.376814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:31.376814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:31.376814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:31.376814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:31.376814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:31.376814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:31.376814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:31.376814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 319/640] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:31.376814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac32f62-cacd-4f36-be05-9c44a99fd2cc None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:31.407973 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7941bf-ca88-4a84-8fb9-f19ad7ea3552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:31.408700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7941bf-ca88-4a84-8fb9-f19ad7ea3552 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:31.409054 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7941bf-ca88-4a84-8fb9-f19ad7ea3552 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:31.409456 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7941bf-ca88-4a84-8fb9-f19ad7ea3552 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:31.410394 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7941bf-ca88-4a84-8fb9-f19ad7ea3552 None admin] RBAC: Adding query filter params (name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:31.410794 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7941bf-ca88-4a84-8fb9-f19ad7ea3552 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:31.412509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7941bf-ca88-4a84-8fb9-f19ad7ea3552 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:31.423489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 322/641] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:15:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:31.424957 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7941bf-ca88-4a84-8fb9-f19ad7ea3552 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:31.481698 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:31.526061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac32f62-cacd-4f36-be05-9c44a99fd2cc None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:31.566382 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:31.583919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50acb7-7bc2-4329-8ae1-8d831efad34d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:31.585058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50acb7-7bc2-4329-8ae1-8d831efad34d None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:31.585398 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50acb7-7bc2-4329-8ae1-8d831efad34d None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:31.585763 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50acb7-7bc2-4329-8ae1-8d831efad34d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:31.585926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7941bf-ca88-4a84-8fb9-f19ad7ea3552 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:31.599125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e50acb7-7bc2-4329-8ae1-8d831efad34d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:31.600727 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e50acb7-7bc2-4329-8ae1-8d831efad34d None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:31.619979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-3e50acb7-7bc2-4329-8ae1-8d831efad34d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:31.619979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:31.619979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:31.619979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:31.619979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:31.619979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:31.619979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:31.619979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:31.619979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:31.619979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:31.619979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:31.619979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:31.619979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:31.619979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:31.619979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:31.619979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:31.621928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:31.621928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:31.621928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:31.621928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:31.621928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:31.621928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:31.621928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:31.621928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:31.621928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:31.621928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:31.621928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:31.621928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:31.621928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:31.621928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:31.621928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:31.621928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:31.621928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:31.623787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:31.623787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:31.623787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:31.623787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 320/642] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:31.623787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50acb7-7bc2-4329-8ae1-8d831efad34d None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:31.641366 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999e895b-0ae2-4c46-8d17-574d369491a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:31.641985 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999e895b-0ae2-4c46-8d17-574d369491a4 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:31.642258 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999e895b-0ae2-4c46-8d17-574d369491a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:31.642524 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999e895b-0ae2-4c46-8d17-574d369491a4 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:31.643232 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999e895b-0ae2-4c46-8d17-574d369491a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:31.643530 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999e895b-0ae2-4c46-8d17-574d369491a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:31.645004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999e895b-0ae2-4c46-8d17-574d369491a4 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:31.667397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 323/643] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:31.667397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999e895b-0ae2-4c46-8d17-574d369491a4 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:31.767556 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50acb7-7bc2-4329-8ae1-8d831efad34d None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:31.810236 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:31.819945 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999e895b-0ae2-4c46-8d17-574d369491a4 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:31.821356 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e0cdbb-4f99-4b5c-b3b9-8a232fe4fbad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:31.822167 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e0cdbb-4f99-4b5c-b3b9-8a232fe4fbad None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:31.822537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e0cdbb-4f99-4b5c-b3b9-8a232fe4fbad None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:31.822967 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e0cdbb-4f99-4b5c-b3b9-8a232fe4fbad None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:31.823612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e0cdbb-4f99-4b5c-b3b9-8a232fe4fbad None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:31.823978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e0cdbb-4f99-4b5c-b3b9-8a232fe4fbad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:31.825687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e0cdbb-4f99-4b5c-b3b9-8a232fe4fbad None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:31.844289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 321/644] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:31.845860 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e0cdbb-4f99-4b5c-b3b9-8a232fe4fbad None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:31.869200 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60d2491-12a2-4c42-8328-0eed9784b457 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:31.870156 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60d2491-12a2-4c42-8328-0eed9784b457 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:31.870598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60d2491-12a2-4c42-8328-0eed9784b457 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:31.871036 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60d2491-12a2-4c42-8328-0eed9784b457 None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:31.872167 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60d2491-12a2-4c42-8328-0eed9784b457 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:31.872689 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60d2491-12a2-4c42-8328-0eed9784b457 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:31.875664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60d2491-12a2-4c42-8328-0eed9784b457 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:31.890511 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 324/645] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:31.891985 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60d2491-12a2-4c42-8328-0eed9784b457 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:31.900616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:31.992229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e0cdbb-4f99-4b5c-b3b9-8a232fe4fbad None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:32.038610 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438391b9-5cb9-4f56-8a1a-31e2de224f20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:32.039270 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438391b9-5cb9-4f56-8a1a-31e2de224f20 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:32.039549 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438391b9-5cb9-4f56-8a1a-31e2de224f20 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:32.039826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438391b9-5cb9-4f56-8a1a-31e2de224f20 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:32.055789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60d2491-12a2-4c42-8328-0eed9784b457 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:32.059021 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438391b9-5cb9-4f56-8a1a-31e2de224f20 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:32.060742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438391b9-5cb9-4f56-8a1a-31e2de224f20 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:32.077768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-438391b9-5cb9-4f56-8a1a-31e2de224f20 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:32.077768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:32.077768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:32.077768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:32.077768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:32.077768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:32.077768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:32.077768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:32.077768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:32.077768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:32.077768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:32.077768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:32.077768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:32.077768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:32.077768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:32.077768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:32.079619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:32.079619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:32.079619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:32.079619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:32.079619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:32.079619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:32.079619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:32.079619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:32.079619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:32.079619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:32.079619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:32.079619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:32.079619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:32.079619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:32.079619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:32.079619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:32.079619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:32.081572 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:32.081572 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:32.081572 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:32.083686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 322/646] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:32.084703 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438391b9-5cb9-4f56-8a1a-31e2de224f20 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:32.118828 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25814493-f521-4b46-8f62-7cf3a1009bdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:32.119876 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25814493-f521-4b46-8f62-7cf3a1009bdf None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:32.120271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25814493-f521-4b46-8f62-7cf3a1009bdf None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:32.121080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25814493-f521-4b46-8f62-7cf3a1009bdf None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:32.122129 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25814493-f521-4b46-8f62-7cf3a1009bdf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:32.122560 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25814493-f521-4b46-8f62-7cf3a1009bdf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:32.124125 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25814493-f521-4b46-8f62-7cf3a1009bdf None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:32.138768 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 325/647] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:32.139420 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25814493-f521-4b46-8f62-7cf3a1009bdf None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:32.168672 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:32.205045 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:32.219715 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438391b9-5cb9-4f56-8a1a-31e2de224f20 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:32.264036 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb6198f-c929-4380-88a2-4f349a7976c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:32.265079 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb6198f-c929-4380-88a2-4f349a7976c8 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:32.267321 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb6198f-c929-4380-88a2-4f349a7976c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:32.267818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb6198f-c929-4380-88a2-4f349a7976c8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:32.280611 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb6198f-c929-4380-88a2-4f349a7976c8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:32.281561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25814493-f521-4b46-8f62-7cf3a1009bdf None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:32.285146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb6198f-c929-4380-88a2-4f349a7976c8 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:32.304007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-cbb6198f-c929-4380-88a2-4f349a7976c8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:32.304007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:32.304007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:32.304007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:32.304007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:32.304007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:32.304007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:32.304007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:32.304007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:32.304007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:32.304007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:32.304007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:32.304007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:32.304007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:32.304007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:32.305790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:32.305790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:32.305790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:32.305790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:32.305790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:32.305790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:32.305790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:32.305790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:32.305790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:32.305790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:32.305790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:32.305790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:32.305790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:32.305790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:32.305790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:32.305790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:32.305790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:32.307590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:32.307590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:32.307590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:32.307590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:32.307590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:32.307590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:32.307590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:32.307590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:32.307590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 323/648] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:32.308609 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb6198f-c929-4380-88a2-4f349a7976c8 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:32.330262 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83979fb9-ee27-402d-8d48-8a39953ae45b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:32.331147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83979fb9-ee27-402d-8d48-8a39953ae45b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:32.331493 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83979fb9-ee27-402d-8d48-8a39953ae45b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:32.331829 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83979fb9-ee27-402d-8d48-8a39953ae45b None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:32.341763 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83979fb9-ee27-402d-8d48-8a39953ae45b None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:32.344501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83979fb9-ee27-402d-8d48-8a39953ae45b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:32.357409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-83979fb9-ee27-402d-8d48-8a39953ae45b None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 14:15:32.357409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:32.357409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:32.357409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:32.357409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:32.357409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:32.357409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:32.357409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:32.357409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:32.357409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:32.357409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:32.357409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:32.357409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:32.357409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:32.357409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:32.357409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:32.359186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:32.361093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:32.361093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:32.361093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:32.361093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:32.361093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:32.361093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:32.361093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:32.361093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:32.361093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 14:15:32.361093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:32.361093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 326/649] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:15:32 2022] GET /identity/v3/users/cinder => generated 84 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:32.361093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83979fb9-ee27-402d-8d48-8a39953ae45b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:32.455554 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:32.460964 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb6198f-c929-4380-88a2-4f349a7976c8 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:32.493225 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83979fb9-ee27-402d-8d48-8a39953ae45b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:32.511016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786b7bc8-b527-4543-a5f8-43f60cbed224 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:32.511991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786b7bc8-b527-4543-a5f8-43f60cbed224 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:32.512387 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786b7bc8-b527-4543-a5f8-43f60cbed224 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:32.512740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786b7bc8-b527-4543-a5f8-43f60cbed224 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:32.513707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786b7bc8-b527-4543-a5f8-43f60cbed224 None admin] RBAC: Adding query filter params (role.id=3ebe3c0be8c34c5eae533311e9b44176, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=35a5a4ef40204e158649ca25583bbde7) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:32.514153 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786b7bc8-b527-4543-a5f8-43f60cbed224 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:32.515830 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786b7bc8-b527-4543-a5f8-43f60cbed224 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:32.522974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 324/650] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:15:32 2022] GET /identity/v3/role_assignments?user.id=35a5a4ef40204e158649ca25583bbde7&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=3ebe3c0be8c34c5eae533311e9b44176 => generated 608 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:32.524501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786b7bc8-b527-4543-a5f8-43f60cbed224 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:32.540882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845798dc-0070-4317-a615-da7a5461b1bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:32.542011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845798dc-0070-4317-a615-da7a5461b1bf None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:32.542538 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845798dc-0070-4317-a615-da7a5461b1bf None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:32.542989 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845798dc-0070-4317-a615-da7a5461b1bf None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:32.544639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845798dc-0070-4317-a615-da7a5461b1bf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:32.545214 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845798dc-0070-4317-a615-da7a5461b1bf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:32.548077 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845798dc-0070-4317-a615-da7a5461b1bf None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:32.551901 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:32.573846 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 327/651] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:32.573846 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845798dc-0070-4317-a615-da7a5461b1bf None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:32.738719 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845798dc-0070-4317-a615-da7a5461b1bf None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:32.741427 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786b7bc8-b527-4543-a5f8-43f60cbed224 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:32.792370 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151677c0-3411-4786-9648-51bdc599089f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:32.793441 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151677c0-3411-4786-9648-51bdc599089f None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:32.793638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151677c0-3411-4786-9648-51bdc599089f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:32.793927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151677c0-3411-4786-9648-51bdc599089f None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:32.803467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151677c0-3411-4786-9648-51bdc599089f None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:32.805030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151677c0-3411-4786-9648-51bdc599089f None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:32.811909 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b93f2e-220a-45ad-a728-ade02150ebfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:32.813628 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b93f2e-220a-45ad-a728-ade02150ebfa None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:32.814090 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b93f2e-220a-45ad-a728-ade02150ebfa None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:32.814550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b93f2e-220a-45ad-a728-ade02150ebfa None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:32.815875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b93f2e-220a-45ad-a728-ade02150ebfa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:32.816571 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b93f2e-220a-45ad-a728-ade02150ebfa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:32.819237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-151677c0-3411-4786-9648-51bdc599089f None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 14:15:32.819237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:32.819237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:32.819237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:32.819237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:32.819237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:32.819237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:32.819237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:32.819237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:32.819237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:32.819237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:32.819237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:32.819237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:32.819237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:32.819237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:32.819237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:32.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:32.821584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:32.821584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:32.821584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:32.821584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:32.821584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:32.821584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:32.821584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:32.821584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:32.821584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 14:15:32.821584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:32.822139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b93f2e-220a-45ad-a728-ade02150ebfa None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:32.822217 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 325/652] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:32 2022] GET /identity/v3/users/neutron => generated 85 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:32.822217 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151677c0-3411-4786-9648-51bdc599089f None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:32.824365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:32.848347 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 328/653] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:32.852377 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b93f2e-220a-45ad-a728-ade02150ebfa None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:32.912571 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:32.969857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151677c0-3411-4786-9648-51bdc599089f None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:33.011771 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b93f2e-220a-45ad-a728-ade02150ebfa None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:33.035198 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f58a7b-2135-4c6f-901b-84e3248e03eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:33.036223 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f58a7b-2135-4c6f-901b-84e3248e03eb None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:33.036743 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f58a7b-2135-4c6f-901b-84e3248e03eb None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:33.037204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f58a7b-2135-4c6f-901b-84e3248e03eb None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:33.038514 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f58a7b-2135-4c6f-901b-84e3248e03eb None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:33.039045 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f58a7b-2135-4c6f-901b-84e3248e03eb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:33.041905 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f58a7b-2135-4c6f-901b-84e3248e03eb None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:33.061632 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c5801a-f3be-44ca-8a7e-b1abadbe78dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:33.063052 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c5801a-f3be-44ca-8a7e-b1abadbe78dd None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:33.063473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c5801a-f3be-44ca-8a7e-b1abadbe78dd None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:33.063877 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c5801a-f3be-44ca-8a7e-b1abadbe78dd None admin] PATH_INFO: `/v3/users/nova` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:33.074524 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c5801a-f3be-44ca-8a7e-b1abadbe78dd None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:33.078951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 326/654] 10.222.0.64 () {56 vars in 1137 bytes} [Tue Oct 4 14:15:32 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:33.079178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c5801a-f3be-44ca-8a7e-b1abadbe78dd None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:33.079270 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f58a7b-2135-4c6f-901b-84e3248e03eb None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:33.101342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-e2c5801a-f3be-44ca-8a7e-b1abadbe78dd None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:15:33.101342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:33.101342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:33.101342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:33.101342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:33.101342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:33.101342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:33.101342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:33.101342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:33.101342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:33.101342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:33.101342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:33.101342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:33.101342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:33.101342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:33.101342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:33.103301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:33.105261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:33.105261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:33.105261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:33.105261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:33.105261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:33.105261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:33.105261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:33.105261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:33.105261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:15:33.105261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:33.105261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 329/655] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Oct 4 14:15:32 2022] GET /identity/v3/users/nova => generated 82 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:33.105261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c5801a-f3be-44ca-8a7e-b1abadbe78dd None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:33.223633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:33.252751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f58a7b-2135-4c6f-901b-84e3248e03eb None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:33.299229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c5801a-f3be-44ca-8a7e-b1abadbe78dd None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:33.315211 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:33.326799 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912ee81d-4456-46f9-943d-335b344293da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:33.327880 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912ee81d-4456-46f9-943d-335b344293da None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:33.328410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912ee81d-4456-46f9-943d-335b344293da None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:33.328883 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912ee81d-4456-46f9-943d-335b344293da None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:33.330180 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912ee81d-4456-46f9-943d-335b344293da None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:33.330716 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912ee81d-4456-46f9-943d-335b344293da None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:33.333148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912ee81d-4456-46f9-943d-335b344293da None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:33.343038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0380625b-7237-4aaa-9483-6b976c423b7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:33.343936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0380625b-7237-4aaa-9483-6b976c423b7a None admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:33.344217 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0380625b-7237-4aaa-9483-6b976c423b7a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:33.344534 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0380625b-7237-4aaa-9483-6b976c423b7a None admin] PATH_INFO: `/v3/projects/895ece8b70ef44adadc3416f81a80731/users/5980e353164c4873bcdd2af847e0a1f1/roles/3ebe3c0be8c34c5eae533311e9b44176` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:33.363377 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0380625b-7237-4aaa-9483-6b976c423b7a None admin] RBAC: Authorizing `identity:create_grant(project_id=895ece8b70ef44adadc3416f81a80731, user_id=5980e353164c4873bcdd2af847e0a1f1, role_id=3ebe3c0be8c34c5eae533311e9b44176)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:33.364900 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0380625b-7237-4aaa-9483-6b976c423b7a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:33.371547 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 327/656] 10.222.0.64 () {56 vars in 1139 bytes} [Tue Oct 4 14:15:33 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:33.374117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912ee81d-4456-46f9-943d-335b344293da None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:33.392543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 330/657] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:15:33 2022] PUT /identity/v3/projects/895ece8b70ef44adadc3416f81a80731/users/5980e353164c4873bcdd2af847e0a1f1/roles/3ebe3c0be8c34c5eae533311e9b44176 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:15:33.394290 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0380625b-7237-4aaa-9483-6b976c423b7a None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:33.609033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912ee81d-4456-46f9-943d-335b344293da None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:33.639826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0380625b-7237-4aaa-9483-6b976c423b7a None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:33.643604 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:33.660347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2125c524-24e6-4f80-a8d7-92c9e72840c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:33.661630 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2125c524-24e6-4f80-a8d7-92c9e72840c5 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:33.661988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2125c524-24e6-4f80-a8d7-92c9e72840c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:33.662548 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2125c524-24e6-4f80-a8d7-92c9e72840c5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:33.690045 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee3c452-298f-4fa9-a31f-fa7fd08f3bfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:33.690669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee3c452-298f-4fa9-a31f-fa7fd08f3bfa None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:33.690954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee3c452-298f-4fa9-a31f-fa7fd08f3bfa None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:33.691204 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee3c452-298f-4fa9-a31f-fa7fd08f3bfa None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:33.691968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee3c452-298f-4fa9-a31f-fa7fd08f3bfa None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:33.692270 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee3c452-298f-4fa9-a31f-fa7fd08f3bfa None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:33.693971 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee3c452-298f-4fa9-a31f-fa7fd08f3bfa None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:33.716968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 331/658] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Oct 4 14:15:33 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:33.717907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee3c452-298f-4fa9-a31f-fa7fd08f3bfa None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:33.745305 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:33.763975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2125c524-24e6-4f80-a8d7-92c9e72840c5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:33.766248 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2125c524-24e6-4f80-a8d7-92c9e72840c5 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:33.905775 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 328/659] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:15:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 531 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:15:33.905775 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2125c524-24e6-4f80-a8d7-92c9e72840c5 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:33.956490 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee3c452-298f-4fa9-a31f-fa7fd08f3bfa None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:34.014876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:34.038632 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2125c524-24e6-4f80-a8d7-92c9e72840c5 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:34.041269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80471fed-5f67-4b40-835c-dfcebb733edf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:34.042575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80471fed-5f67-4b40-835c-dfcebb733edf None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:34.043271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80471fed-5f67-4b40-835c-dfcebb733edf None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:34.043944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80471fed-5f67-4b40-835c-dfcebb733edf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:34.086615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:34.112283 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05504dc-5da5-4e5f-8978-dac822fd45ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:34.113894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05504dc-5da5-4e5f-8978-dac822fd45ee None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:34.114605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05504dc-5da5-4e5f-8978-dac822fd45ee None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:34.115286 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05504dc-5da5-4e5f-8978-dac822fd45ee None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:34.156816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80471fed-5f67-4b40-835c-dfcebb733edf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:34.160204 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80471fed-5f67-4b40-835c-dfcebb733edf None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:34.205155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05504dc-5da5-4e5f-8978-dac822fd45ee None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:34.207018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05504dc-5da5-4e5f-8978-dac822fd45ee None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:34.312853 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:34.335137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 329/660] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:15:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 431 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:15:34.336351 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05504dc-5da5-4e5f-8978-dac822fd45ee None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:34.416018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 332/661] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:15:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 698 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:15:34.418105 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80471fed-5f67-4b40-835c-dfcebb733edf None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:34.454907 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05504dc-5da5-4e5f-8978-dac822fd45ee None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:34.495760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320757b8-b499-45d9-893d-30661d999ee0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:34.496595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320757b8-b499-45d9-893d-30661d999ee0 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:34.496980 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320757b8-b499-45d9-893d-30661d999ee0 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:34.497438 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320757b8-b499-45d9-893d-30661d999ee0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:34.509865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320757b8-b499-45d9-893d-30661d999ee0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:34.511514 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320757b8-b499-45d9-893d-30661d999ee0 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:34.530233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-320757b8-b499-45d9-893d-30661d999ee0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:34.530233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:34.530233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:34.530233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:34.530233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:34.530233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:34.530233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:34.530233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:34.530233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:34.530233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:34.530233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:34.530233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:34.530233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:34.530233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:34.530233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:34.530233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:34.533003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:34.533003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:34.533003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:34.533003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:34.533003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:34.533003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:34.533003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:34.533003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:34.533003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:34.533003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:34.533003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:34.533003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:34.533003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:34.533003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:34.533003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:34.533003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:34.533003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:34.535149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:34.535149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:34.535149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:34.535149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 330/662] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:34.535149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320757b8-b499-45d9-893d-30661d999ee0 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:34.576857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:34.589942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:34.655362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80471fed-5f67-4b40-835c-dfcebb733edf None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:34.668911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320757b8-b499-45d9-893d-30661d999ee0 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:34.713073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007a0d48-898a-47ee-9dbd-9e8e4e9890fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:34.713557 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007a0d48-898a-47ee-9dbd-9e8e4e9890fb None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:34.714037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007a0d48-898a-47ee-9dbd-9e8e4e9890fb None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:34.714430 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007a0d48-898a-47ee-9dbd-9e8e4e9890fb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:34.728726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007a0d48-898a-47ee-9dbd-9e8e4e9890fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:34.729171 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd4cbdf-bb54-4d0b-b152-b9f17d234d8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:34.730664 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007a0d48-898a-47ee-9dbd-9e8e4e9890fb None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:34.730887 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd4cbdf-bb54-4d0b-b152-b9f17d234d8c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:34.731571 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd4cbdf-bb54-4d0b-b152-b9f17d234d8c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:34.732247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd4cbdf-bb54-4d0b-b152-b9f17d234d8c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:34.748976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-007a0d48-898a-47ee-9dbd-9e8e4e9890fb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:34.748976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:34.748976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:34.748976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:34.748976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:34.748976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:34.748976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:34.748976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:34.748976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:34.748976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:34.748976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:34.748976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:34.748976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:34.748976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:34.748976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:34.748976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:34.752550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:34.752550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:34.752550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:34.752550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:34.752550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:34.752550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:34.752550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:34.752550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:34.752550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:34.752550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:34.752550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:34.752550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:34.752550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:34.752550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:34.752550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:34.752550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:34.752550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:34.754310 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:34.754310 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:34.754310 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:34.754310 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 331/663] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:34.754310 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007a0d48-898a-47ee-9dbd-9e8e4e9890fb None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:34.760807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd4cbdf-bb54-4d0b-b152-b9f17d234d8c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:34.763803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd4cbdf-bb54-4d0b-b152-b9f17d234d8c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:34.795814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-ffd4cbdf-bb54-4d0b-b152-b9f17d234d8c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:34.795814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:34.795814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:34.795814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:34.795814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:34.795814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:34.795814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:34.795814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:34.795814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:34.795814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:34.795814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:34.795814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:34.795814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:34.795814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:34.795814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:34.795814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:34.797784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:34.797784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:34.797784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:34.797784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:34.797784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:34.797784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:34.797784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:34.797784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:34.797784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:34.797784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:34.797784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:34.797784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:34.797784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:34.797784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:34.797784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:34.797784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:34.797784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:34.799599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:34.799599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:34.799599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:34.803400 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 333/664] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 386 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:34.805466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd4cbdf-bb54-4d0b-b152-b9f17d234d8c None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:34.875918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:34.888651 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007a0d48-898a-47ee-9dbd-9e8e4e9890fb None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:34.933257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c9802d-59d8-4519-81d8-ed056737b104 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:34.933527 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c9802d-59d8-4519-81d8-ed056737b104 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:34.933857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c9802d-59d8-4519-81d8-ed056737b104 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:34.934125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c9802d-59d8-4519-81d8-ed056737b104 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:34.934798 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c9802d-59d8-4519-81d8-ed056737b104 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:34.935104 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c9802d-59d8-4519-81d8-ed056737b104 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:34.936658 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c9802d-59d8-4519-81d8-ed056737b104 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:34.960367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 332/665] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:34.960367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c9802d-59d8-4519-81d8-ed056737b104 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:35.019179 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd4cbdf-bb54-4d0b-b152-b9f17d234d8c None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:35.051000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:35.082475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c9802d-59d8-4519-81d8-ed056737b104 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:35.100081 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ada64a-a973-4239-b46f-6509dccf4df3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:35.101759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ada64a-a973-4239-b46f-6509dccf4df3 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:35.102613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ada64a-a973-4239-b46f-6509dccf4df3 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:35.103359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ada64a-a973-4239-b46f-6509dccf4df3 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:35.105628 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ada64a-a973-4239-b46f-6509dccf4df3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:35.106475 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ada64a-a973-4239-b46f-6509dccf4df3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:35.110657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ada64a-a973-4239-b46f-6509dccf4df3 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:35.126935 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31aba205-53ef-4df4-8c1c-64f9a4ebe603 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:35.128461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aba205-53ef-4df4-8c1c-64f9a4ebe603 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:35.128461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aba205-53ef-4df4-8c1c-64f9a4ebe603 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:35.128461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aba205-53ef-4df4-8c1c-64f9a4ebe603 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:35.129139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31aba205-53ef-4df4-8c1c-64f9a4ebe603 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:35.129514 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31aba205-53ef-4df4-8c1c-64f9a4ebe603 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:35.131112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31aba205-53ef-4df4-8c1c-64f9a4ebe603 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:35.142169 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 334/666] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 338 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:35.145135 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ada64a-a973-4239-b46f-6509dccf4df3 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:35.157669 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 333/667] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:35.159411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31aba205-53ef-4df4-8c1c-64f9a4ebe603 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:35.190450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:35.355451 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31aba205-53ef-4df4-8c1c-64f9a4ebe603 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:35.399855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ada64a-a973-4239-b46f-6509dccf4df3 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:35.422227 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae8a0bd-9a5d-4a90-b9e0-e8ccca5d3fd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:35.423944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae8a0bd-9a5d-4a90-b9e0-e8ccca5d3fd5 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:35.424462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae8a0bd-9a5d-4a90-b9e0-e8ccca5d3fd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:35.425340 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae8a0bd-9a5d-4a90-b9e0-e8ccca5d3fd5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:35.443894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae8a0bd-9a5d-4a90-b9e0-e8ccca5d3fd5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:35.446917 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae8a0bd-9a5d-4a90-b9e0-e8ccca5d3fd5 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:35.451954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43cb2e8-1330-4645-a5c2-7d8ea649dd66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:35.451954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43cb2e8-1330-4645-a5c2-7d8ea649dd66 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:35.451954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43cb2e8-1330-4645-a5c2-7d8ea649dd66 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:35.451954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43cb2e8-1330-4645-a5c2-7d8ea649dd66 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:35.468347 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43cb2e8-1330-4645-a5c2-7d8ea649dd66 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:35.468347 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43cb2e8-1330-4645-a5c2-7d8ea649dd66 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:35.473568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-2ae8a0bd-9a5d-4a90-b9e0-e8ccca5d3fd5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:35.473568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:35.473568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:35.473568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:35.473568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:35.473568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:35.473568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:35.473568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:35.473568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:35.473568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:35.473568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:35.473568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:35.473568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:35.473568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:35.473568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:35.475289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:35.475289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:35.475289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:35.475289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:35.475289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:35.475289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:35.475289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:35.475289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:35.475289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:35.475289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:35.475289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:35.475289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:35.475289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:35.475289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:35.475289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:35.475289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:35.475289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:35.477348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:35.477348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:35.477348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:35.477348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:35.477348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:35.477348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:35.477348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:35.477348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:35.478638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-c43cb2e8-1330-4645-a5c2-7d8ea649dd66 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:35.478638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:35.478638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:35.478638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:35.478638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:35.478638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:35.478638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:35.478638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:35.478638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:35.478638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:35.478638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:35.478638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:35.478638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:35.478638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:35.478638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:35.481373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:35.481373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:35.481373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:35.481373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:35.481373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:35.481373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:35.481373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:35.481373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:35.481373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:35.481373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:35.481373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:35.481373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:35.481373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:35.481373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:35.481373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:35.481373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:35.481373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:35.494813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:35.494813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:35.494813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:35.494813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:35.494813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:35.494813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:35.494813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:35.494813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:35.494813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 335/668] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 334 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:35.494813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43cb2e8-1330-4645-a5c2-7d8ea649dd66 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:35.497025 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 334/669] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 325 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:35.497025 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae8a0bd-9a5d-4a90-b9e0-e8ccca5d3fd5 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:35.565033 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:35.621003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:35.660337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43cb2e8-1330-4645-a5c2-7d8ea649dd66 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:35.666907 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae8a0bd-9a5d-4a90-b9e0-e8ccca5d3fd5 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:35.707634 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e02ac24-ee60-45db-aab6-a9d77158cf35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:35.708192 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e02ac24-ee60-45db-aab6-a9d77158cf35 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:35.708543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e02ac24-ee60-45db-aab6-a9d77158cf35 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:35.709052 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e02ac24-ee60-45db-aab6-a9d77158cf35 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:35.717416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad518de-e204-4878-9b5f-78e49ae5e913 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:35.717627 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e02ac24-ee60-45db-aab6-a9d77158cf35 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:35.718602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad518de-e204-4878-9b5f-78e49ae5e913 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:35.719134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad518de-e204-4878-9b5f-78e49ae5e913 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:35.719371 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e02ac24-ee60-45db-aab6-a9d77158cf35 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:35.719810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad518de-e204-4878-9b5f-78e49ae5e913 None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:35.720797 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad518de-e204-4878-9b5f-78e49ae5e913 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:35.721551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad518de-e204-4878-9b5f-78e49ae5e913 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:35.723834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad518de-e204-4878-9b5f-78e49ae5e913 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:35.732823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-2e02ac24-ee60-45db-aab6-a9d77158cf35 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:35.732823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:35.732823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:35.732823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:35.732823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:35.732823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:35.732823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:35.732823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:35.732823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:35.732823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:35.732823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:35.732823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:35.732823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:35.732823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:35.732823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:35.733969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:35.733969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:35.733969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:35.733969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:35.733969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:35.733969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:35.733969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:35.733969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:35.733969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:35.733969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:35.733969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:35.733969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:35.733969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:35.733969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:35.733969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:35.733969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:35.733969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:35.735611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:35.735611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:35.735611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:35.735611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:35.735611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:35.735611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:35.735611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:35.735611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:35.735611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 336/670] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:35.735611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e02ac24-ee60-45db-aab6-a9d77158cf35 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:35.744343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 335/671] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:15:35.746657 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873e4f9a-4557-4a25-ad7f-9efd88ee7a21 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:35.747628 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873e4f9a-4557-4a25-ad7f-9efd88ee7a21 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:35.748108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873e4f9a-4557-4a25-ad7f-9efd88ee7a21 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:35.751923 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 336/672] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:35 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:35.754405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873e4f9a-4557-4a25-ad7f-9efd88ee7a21 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:35.880252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873e4f9a-4557-4a25-ad7f-9efd88ee7a21 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:35.897415 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:35.911019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e02ac24-ee60-45db-aab6-a9d77158cf35 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:35.918707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:35.925492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9c5753-3fdb-45f8-ab76-3309d559ffcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:35.926114 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9c5753-3fdb-45f8-ab76-3309d559ffcf None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:35.930501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9c5753-3fdb-45f8-ab76-3309d559ffcf None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:35.930501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9c5753-3fdb-45f8-ab76-3309d559ffcf None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:35.930501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9c5753-3fdb-45f8-ab76-3309d559ffcf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:35.930501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9c5753-3fdb-45f8-ab76-3309d559ffcf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:35.930501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9c5753-3fdb-45f8-ab76-3309d559ffcf None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:35.944742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 337/673] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:35.945997 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9c5753-3fdb-45f8-ab76-3309d559ffcf None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:35.977996 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f173de-94e7-4acf-b514-d8e79c979ba8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:35.978524 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f173de-94e7-4acf-b514-d8e79c979ba8 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:35.978936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f173de-94e7-4acf-b514-d8e79c979ba8 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:35.979382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f173de-94e7-4acf-b514-d8e79c979ba8 None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:35.979981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f173de-94e7-4acf-b514-d8e79c979ba8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:35.980265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f173de-94e7-4acf-b514-d8e79c979ba8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:35.981741 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f173de-94e7-4acf-b514-d8e79c979ba8 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:35.998119 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 337/674] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:36.004601 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8e63f3-af4e-4b5e-aef7-8363e01662de None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:36.004877 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8e63f3-af4e-4b5e-aef7-8363e01662de None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:36.005016 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8e63f3-af4e-4b5e-aef7-8363e01662de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:36.025053 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-ab8e63f3-af4e-4b5e-aef7-8363e01662de None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:36.026280 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ab8e63f3-af4e-4b5e-aef7-8363e01662de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:36.114986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 338/675] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:36.114986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8e63f3-af4e-4b5e-aef7-8363e01662de None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:36.148611 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9c5753-3fdb-45f8-ab76-3309d559ffcf None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:36.223388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988a108f-b75a-4194-9745-1b9e71010e87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:36.224699 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988a108f-b75a-4194-9745-1b9e71010e87 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:36.226232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988a108f-b75a-4194-9745-1b9e71010e87 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:36.226699 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988a108f-b75a-4194-9745-1b9e71010e87 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:36.227629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988a108f-b75a-4194-9745-1b9e71010e87 None admin] RBAC: Adding query filter params (role.id=3ebe3c0be8c34c5eae533311e9b44176, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=8dbeccfb9a194992959f0b341ef0d4a5) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:36.228215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988a108f-b75a-4194-9745-1b9e71010e87 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:36.229912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988a108f-b75a-4194-9745-1b9e71010e87 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:36.247362 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 338/676] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:15:35 2022] GET /identity/v3/role_assignments?user.id=8dbeccfb9a194992959f0b341ef0d4a5&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=3ebe3c0be8c34c5eae533311e9b44176 => generated 608 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:36.247362 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988a108f-b75a-4194-9745-1b9e71010e87 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:36.268874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:36.302801 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8e63f3-af4e-4b5e-aef7-8363e01662de None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:36.388565 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c24ebe5-97f2-461b-8dbe-258c30a8bd72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:36.389209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c24ebe5-97f2-461b-8dbe-258c30a8bd72 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:36.390219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c24ebe5-97f2-461b-8dbe-258c30a8bd72 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:36.390219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c24ebe5-97f2-461b-8dbe-258c30a8bd72 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:36.390966 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c24ebe5-97f2-461b-8dbe-258c30a8bd72 None admin] RBAC: Adding query filter params (role.id=3ebe3c0be8c34c5eae533311e9b44176, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=03081d7d581745c88c12e9d089fcb2d3) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:36.391273 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c24ebe5-97f2-461b-8dbe-258c30a8bd72 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:36.393088 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c24ebe5-97f2-461b-8dbe-258c30a8bd72 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:36.405166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 339/677] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:15:36 2022] GET /identity/v3/role_assignments?user.id=03081d7d581745c88c12e9d089fcb2d3&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=3ebe3c0be8c34c5eae533311e9b44176 => generated 608 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:36.409194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:36.477170 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988a108f-b75a-4194-9745-1b9e71010e87 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:36.494565 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3048955-c647-4c59-b30f-d8c01a6983bf None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:36.494762 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3048955-c647-4c59-b30f-d8c01a6983bf None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:36.495025 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3048955-c647-4c59-b30f-d8c01a6983bf None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:36.500352 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 340/678] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:15:36.503729 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd6da64-b929-4b69-828f-8a27a6986972 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:36.504038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd6da64-b929-4b69-828f-8a27a6986972 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:36.504297 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd6da64-b929-4b69-828f-8a27a6986972 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:36.527878 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-0dd6da64-b929-4b69-828f-8a27a6986972 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:36.529245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-0dd6da64-b929-4b69-828f-8a27a6986972 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:36.545298 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2389bc72-6780-41dd-bf1f-28b8ad41d1d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:36.547096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2389bc72-6780-41dd-bf1f-28b8ad41d1d4 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:36.547658 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2389bc72-6780-41dd-bf1f-28b8ad41d1d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:36.548151 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2389bc72-6780-41dd-bf1f-28b8ad41d1d4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:36.549212 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2389bc72-6780-41dd-bf1f-28b8ad41d1d4 None admin] RBAC: Adding query filter params (role.id=3ebe3c0be8c34c5eae533311e9b44176, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=058c027ade94498fbab7b9347e87c9c3) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:36.549549 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2389bc72-6780-41dd-bf1f-28b8ad41d1d4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:36.551072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2389bc72-6780-41dd-bf1f-28b8ad41d1d4 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:36.561380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 339/679] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:15:36 2022] GET /identity/v3/role_assignments?user.id=058c027ade94498fbab7b9347e87c9c3&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=3ebe3c0be8c34c5eae533311e9b44176 => generated 608 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:36.629354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 341/680] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:15:36.674728 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2389bc72-6780-41dd-bf1f-28b8ad41d1d4 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:36.885310 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2389bc72-6780-41dd-bf1f-28b8ad41d1d4 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:36.927263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:36.940141 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd6da64-b929-4b69-828f-8a27a6986972 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:36.954628 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff86723-a076-4b29-aaf9-d914019d8246 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:36.956035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff86723-a076-4b29-aaf9-d914019d8246 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:36.956740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff86723-a076-4b29-aaf9-d914019d8246 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:36.957377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff86723-a076-4b29-aaf9-d914019d8246 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:36.980439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff86723-a076-4b29-aaf9-d914019d8246 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:36.983281 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff86723-a076-4b29-aaf9-d914019d8246 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:37.011462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-fff86723-a076-4b29-aaf9-d914019d8246 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:37.011462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:37.011462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:37.011462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:37.011462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:37.011462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:37.011462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:37.011462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:37.011462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:37.011462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:37.011462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:37.011462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:37.011462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:37.011462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:37.011462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:37.011462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:37.013659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:37.013659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:37.013659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:37.013659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:37.013659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:37.013659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:37.013659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:37.013659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:37.013659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:37.013659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:37.013659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:37.013659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:37.013659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:37.013659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:37.013659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:37.013659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:37.013659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:37.016350 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:37.016350 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:37.016350 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:37.019268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 340/681] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 346 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:37.031270 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff86723-a076-4b29-aaf9-d914019d8246 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:37.110456 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:37.158370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd6da64-b929-4b69-828f-8a27a6986972 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:37.233142 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb751fb-880b-4e2e-9a46-cc0e74fcdfe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:37.234424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb751fb-880b-4e2e-9a46-cc0e74fcdfe0 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:37.234705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb751fb-880b-4e2e-9a46-cc0e74fcdfe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:37.234981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb751fb-880b-4e2e-9a46-cc0e74fcdfe0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:37.270092 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb751fb-880b-4e2e-9a46-cc0e74fcdfe0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:37.282950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff86723-a076-4b29-aaf9-d914019d8246 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:37.290471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-8cb751fb-880b-4e2e-9a46-cc0e74fcdfe0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:37.290471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:37.290471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:37.290471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:37.290471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:37.290471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:37.290471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:37.290471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:37.290471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:37.290471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:37.290471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:37.290471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:37.290471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:37.290471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:37.290471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:37.290471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:37.292265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:37.292265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:37.292265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:37.292265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:37.292265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:37.292265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:37.292265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:37.292265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:37.292265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:37.292265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:37.292265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:37.292265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:37.292265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:37.292265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:37.292265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:37.292265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:37.293981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:37.293981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:37.293981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:37.293981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:37.293981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:37.293981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:37.293981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:37.293981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:37.293981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:37.293981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:37.293981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:37.293981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:37.297316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 342/682] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:36 2022] GET /identity/v3/roles/service => generated 85 bytes in 355 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:37.301976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb751fb-880b-4e2e-9a46-cc0e74fcdfe0 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:37.400900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5762de-8718-4a8e-bb14-56806122a4a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:37.401532 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5762de-8718-4a8e-bb14-56806122a4a7 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:37.401815 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5762de-8718-4a8e-bb14-56806122a4a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:37.402090 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5762de-8718-4a8e-bb14-56806122a4a7 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:37.402747 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5762de-8718-4a8e-bb14-56806122a4a7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:37.403050 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5762de-8718-4a8e-bb14-56806122a4a7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:37.404484 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5762de-8718-4a8e-bb14-56806122a4a7 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:37.430977 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 341/683] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 401 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:15:37.439672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5762de-8718-4a8e-bb14-56806122a4a7 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:37.514398 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:37.552209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb751fb-880b-4e2e-9a46-cc0e74fcdfe0 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:37.659340 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a42682-5d46-4448-9162-da3d23b098de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:37.663768 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a42682-5d46-4448-9162-da3d23b098de None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:37.664161 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a42682-5d46-4448-9162-da3d23b098de None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:37.664386 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a42682-5d46-4448-9162-da3d23b098de None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:37.665040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a42682-5d46-4448-9162-da3d23b098de None admin] RBAC: Adding query filter params (name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:37.665346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a42682-5d46-4448-9162-da3d23b098de None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:37.666794 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a42682-5d46-4448-9162-da3d23b098de None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:37.679694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:37.692667 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 343/684] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:15:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 391 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:37.694233 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a42682-5d46-4448-9162-da3d23b098de None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:37.696716 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5762de-8718-4a8e-bb14-56806122a4a7 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:37.759067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fcbcdd-3dc1-4932-874e-8fcaaecbd5aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:37.759709 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fcbcdd-3dc1-4932-874e-8fcaaecbd5aa None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:37.759998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fcbcdd-3dc1-4932-874e-8fcaaecbd5aa None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:37.760261 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fcbcdd-3dc1-4932-874e-8fcaaecbd5aa None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:37.761046 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fcbcdd-3dc1-4932-874e-8fcaaecbd5aa None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:37.762580 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fcbcdd-3dc1-4932-874e-8fcaaecbd5aa None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:37.804012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 342/685] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:15:37 2022] POST /identity/v3/users => generated 254 bytes in 365 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:15:37.932357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a42682-5d46-4448-9162-da3d23b098de None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:38.001369 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f744d35-ef80-4d7e-84d5-bc10ca0992fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:38.002293 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f744d35-ef80-4d7e-84d5-bc10ca0992fa None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:38.002293 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f744d35-ef80-4d7e-84d5-bc10ca0992fa None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:38.002600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f744d35-ef80-4d7e-84d5-bc10ca0992fa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:38.027262 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f744d35-ef80-4d7e-84d5-bc10ca0992fa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:38.029975 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f744d35-ef80-4d7e-84d5-bc10ca0992fa None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:38.053436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-2f744d35-ef80-4d7e-84d5-bc10ca0992fa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:38.053436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:38.053436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:38.053436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:38.053436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:38.053436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:38.053436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:38.053436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:38.053436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:38.053436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:38.053436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:38.053436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:38.053436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:38.053436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:38.053436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:38.053436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:38.055863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:38.055863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:38.055863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:38.055863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:38.055863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:38.055863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:38.055863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:38.055863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:38.055863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:38.055863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:38.055863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:38.055863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:38.055863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:38.055863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:38.055863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:38.055863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:38.055863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:38.057645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:38.057645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:38.057645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:38.057645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 344/686] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 361 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:38.061461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fcbcdd-3dc1-4932-874e-8fcaaecbd5aa None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:38.197153 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:38.212879 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fcbcdd-3dc1-4932-874e-8fcaaecbd5aa None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:38.261227 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe89216-096e-4cd9-8628-3308e6352b44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:38.261759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe89216-096e-4cd9-8628-3308e6352b44 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:38.262041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe89216-096e-4cd9-8628-3308e6352b44 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:38.262316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe89216-096e-4cd9-8628-3308e6352b44 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:38.262980 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe89216-096e-4cd9-8628-3308e6352b44 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:38.263283 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe89216-096e-4cd9-8628-3308e6352b44 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:38.265314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe89216-096e-4cd9-8628-3308e6352b44 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:38.292213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 343/687] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:15:38.298354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f744d35-ef80-4d7e-84d5-bc10ca0992fa None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:38.428295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:38.543240 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f744d35-ef80-4d7e-84d5-bc10ca0992fa None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:38.585601 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2429c154-a122-446a-84cb-921ab7346a02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:38.586262 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2429c154-a122-446a-84cb-921ab7346a02 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:38.586542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2429c154-a122-446a-84cb-921ab7346a02 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:38.586814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2429c154-a122-446a-84cb-921ab7346a02 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:38.596041 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2429c154-a122-446a-84cb-921ab7346a02 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:38.597555 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2429c154-a122-446a-84cb-921ab7346a02 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:38.610964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-2429c154-a122-446a-84cb-921ab7346a02 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 14:15:38.610964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:38.610964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:38.610964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:38.610964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:38.610964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:38.610964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:38.610964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:38.610964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:38.610964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:38.610964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:38.610964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:38.610964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:38.610964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:38.610964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:38.610964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:38.612760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:38.614987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:38.614987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:38.614987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:38.614987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:38.614987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:38.614987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:38.614987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:38.614987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:38.614987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 14:15:38.614987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:38.614987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 345/688] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:15:38 2022] GET /identity/v3/users/swift => generated 83 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:38.622073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe89216-096e-4cd9-8628-3308e6352b44 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:38.788055 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe89216-096e-4cd9-8628-3308e6352b44 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:38.840236 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6ad258-0c66-4f26-aff8-16d2380a89e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:38.840796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6ad258-0c66-4f26-aff8-16d2380a89e8 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:38.841076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6ad258-0c66-4f26-aff8-16d2380a89e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:38.841357 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6ad258-0c66-4f26-aff8-16d2380a89e8 None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:38.842103 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6ad258-0c66-4f26-aff8-16d2380a89e8 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:38.842415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6ad258-0c66-4f26-aff8-16d2380a89e8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:38.843956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6ad258-0c66-4f26-aff8-16d2380a89e8 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:38.866764 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 344/689] 10.222.0.64 () {56 vars in 1135 bytes} [Tue Oct 4 14:15:38 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:38.873458 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2429c154-a122-446a-84cb-921ab7346a02 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:39.167338 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2429c154-a122-446a-84cb-921ab7346a02 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:39.187616 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:39.223288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82dc8deb-6319-42fa-af86-68d1e51de853 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:39.224031 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dc8deb-6319-42fa-af86-68d1e51de853 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:39.224335 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dc8deb-6319-42fa-af86-68d1e51de853 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:39.224615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dc8deb-6319-42fa-af86-68d1e51de853 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:39.343496 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82dc8deb-6319-42fa-af86-68d1e51de853 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:39.343496 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82dc8deb-6319-42fa-af86-68d1e51de853 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:39.571884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 346/690] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:15:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 699 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:15:39.581039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6ad258-0c66-4f26-aff8-16d2380a89e8 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:39.597730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:39.785196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6ad258-0c66-4f26-aff8-16d2380a89e8 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:39.847275 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e3b499-d06f-413f-8218-827ddcd010af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:39.848201 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e3b499-d06f-413f-8218-827ddcd010af None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:39.848654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e3b499-d06f-413f-8218-827ddcd010af None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:39.849104 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e3b499-d06f-413f-8218-827ddcd010af None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:39.861714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e3b499-d06f-413f-8218-827ddcd010af None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:39.863448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e3b499-d06f-413f-8218-827ddcd010af None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:39.887670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-18e3b499-d06f-413f-8218-827ddcd010af None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:39.887670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:39.887670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:39.887670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:39.887670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:39.887670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:39.887670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:39.887670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:39.887670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:39.887670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:39.887670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:39.887670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:39.887670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:39.887670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:39.887670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:39.887670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:39.888991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:39.888991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:39.888991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:39.888991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:39.888991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:39.888991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:39.888991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:39.888991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:39.888991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:39.888991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:39.888991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:39.888991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:39.888991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:39.888991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:39.888991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:39.888991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:39.888991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:39.890247 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:39.890247 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:39.890247 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:39.893571 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 345/691] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:39.901425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82dc8deb-6319-42fa-af86-68d1e51de853 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:39.964866 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:40.024203 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76277499-0f85-4edc-b158-436c2b544f54 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:40.024493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76277499-0f85-4edc-b158-436c2b544f54 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:40.024663 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76277499-0f85-4edc-b158-436c2b544f54 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:40.032401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 346/692] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:40 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:40.039966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7136d1-d6df-4abf-9895-083c97a69554 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:40.040199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7136d1-d6df-4abf-9895-083c97a69554 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:40.040529 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7136d1-d6df-4abf-9895-083c97a69554 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:40.072610 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-4a7136d1-d6df-4abf-9895-083c97a69554 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:40.078661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-4a7136d1-d6df-4abf-9895-083c97a69554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:40.172908 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82dc8deb-6319-42fa-af86-68d1e51de853 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:40.242584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:40.254655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22811b79-a336-4a50-b1b0-d802643b63b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:40.255735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22811b79-a336-4a50-b1b0-d802643b63b9 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:40.256165 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22811b79-a336-4a50-b1b0-d802643b63b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:40.256812 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22811b79-a336-4a50-b1b0-d802643b63b9 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:40.257839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22811b79-a336-4a50-b1b0-d802643b63b9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:40.258394 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22811b79-a336-4a50-b1b0-d802643b63b9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:40.261187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22811b79-a336-4a50-b1b0-d802643b63b9 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:40.268896 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 347/693] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:40.270625 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6427a6-eae5-4c32-aae2-630201b9b8aa None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:40.271171 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6427a6-eae5-4c32-aae2-630201b9b8aa None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:40.271648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6427a6-eae5-4c32-aae2-630201b9b8aa None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:40.277548 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 348/694] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:40.279310 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7978484c-2774-4551-b664-be8e5720b857 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:40.280079 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7978484c-2774-4551-b664-be8e5720b857 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:40.281354 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7978484c-2774-4551-b664-be8e5720b857 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:40.298816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 347/695] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 399 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:40.307128 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22811b79-a336-4a50-b1b0-d802643b63b9 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:40.322300 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-7978484c-2774-4551-b664-be8e5720b857 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:40.324546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7978484c-2774-4551-b664-be8e5720b857 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:40.411915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:40.431783 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 349/696] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:40.431783 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc0b131-ee4f-4752-b248-6ed77a2b3765 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:40.431783 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc0b131-ee4f-4752-b248-6ed77a2b3765 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:40.431783 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc0b131-ee4f-4752-b248-6ed77a2b3765 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:40.442689 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 350/697] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:40 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:40.445021 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c992c8-3d17-46db-8b91-e164dda6b1fa None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:40.445314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c992c8-3d17-46db-8b91-e164dda6b1fa None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:40.445592 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c992c8-3d17-46db-8b91-e164dda6b1fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:40.493984 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-14c992c8-3d17-46db-8b91-e164dda6b1fa None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:40.495265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-14c992c8-3d17-46db-8b91-e164dda6b1fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:40.499222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22811b79-a336-4a50-b1b0-d802643b63b9 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:40.572420 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10142ca-5c50-44a9-9eca-177996745ca9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:40.574046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10142ca-5c50-44a9-9eca-177996745ca9 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:40.575489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10142ca-5c50-44a9-9eca-177996745ca9 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:40.576347 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10142ca-5c50-44a9-9eca-177996745ca9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:40.595502 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10142ca-5c50-44a9-9eca-177996745ca9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:40.598809 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10142ca-5c50-44a9-9eca-177996745ca9 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:40.647653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-e10142ca-5c50-44a9-9eca-177996745ca9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:40.647653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:40.647653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:40.647653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:40.647653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:40.647653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:40.647653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:40.647653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:40.647653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:40.647653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:40.647653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:40.647653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:40.647653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:40.647653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:40.647653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:40.649495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:40.649495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:40.649495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:40.649495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:40.649495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:40.649495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:40.649495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:40.649495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:40.649495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:40.649495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:40.649495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:40.649495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:40.649495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:40.649495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:40.649495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:40.649495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:40.649495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:40.651345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:40.651345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:40.651345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:40.651345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:40.651345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:40.651345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:40.651345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:40.651345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:40.657521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 348/698] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 351 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:40.666691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10142ca-5c50-44a9-9eca-177996745ca9 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:40.674015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 351/699] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 14:15:40.757845 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c992c8-3d17-46db-8b91-e164dda6b1fa None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:40.830545 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:40.894001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c992c8-3d17-46db-8b91-e164dda6b1fa None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:40.923559 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10142ca-5c50-44a9-9eca-177996745ca9 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:40.935009 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3e00d0-aa90-4ee3-a6d8-bbe6bba0fed0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:40.935671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3e00d0-aa90-4ee3-a6d8-bbe6bba0fed0 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:40.935950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3e00d0-aa90-4ee3-a6d8-bbe6bba0fed0 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:40.936225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3e00d0-aa90-4ee3-a6d8-bbe6bba0fed0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:40.949525 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3e00d0-aa90-4ee3-a6d8-bbe6bba0fed0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:40.955410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-2e3e00d0-aa90-4ee3-a6d8-bbe6bba0fed0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:15:40.955410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:40.955410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:40.955410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:40.955410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:40.955410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:40.955410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:40.955410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:40.955410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:40.955410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:40.955410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:40.955410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:40.955410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:40.955410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:40.955410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:40.955410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:40.957108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:40.957108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:40.957108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:40.957108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:40.957108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:40.957108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:40.957108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:40.957108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:40.957108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:40.957108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:40.957108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:40.957108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:40.957108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:40.957108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:40.957108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:40.957108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:40.958782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:40.958782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:40.958782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:40.958782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:40.958782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:40.958782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:40.958782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:40.958782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:40.958782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:40.958782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:40.958782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:15:40.958782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:40.961622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 352/700] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:15:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:40.962661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3e00d0-aa90-4ee3-a6d8-bbe6bba0fed0 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:40.998268 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:41.013762 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ffe1d8-d649-461b-b01a-4253a049a822 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:41.020599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ffe1d8-d649-461b-b01a-4253a049a822 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:41.020599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ffe1d8-d649-461b-b01a-4253a049a822 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:41.020599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ffe1d8-d649-461b-b01a-4253a049a822 None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:41.020599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ffe1d8-d649-461b-b01a-4253a049a822 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:41.020599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ffe1d8-d649-461b-b01a-4253a049a822 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:41.022554 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ffe1d8-d649-461b-b01a-4253a049a822 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:41.060050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 349/701] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 394 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:41.062410 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ffe1d8-d649-461b-b01a-4253a049a822 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:41.090999 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3e00d0-aa90-4ee3-a6d8-bbe6bba0fed0 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:41.102792 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:41.145037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace95297-0c31-45f3-8887-ce01eb29780f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:41.146452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace95297-0c31-45f3-8887-ce01eb29780f None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:41.147135 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace95297-0c31-45f3-8887-ce01eb29780f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:41.147862 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace95297-0c31-45f3-8887-ce01eb29780f None admin] PATH_INFO: `/v3/services/network` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:41.149123 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace95297-0c31-45f3-8887-ce01eb29780f None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:41.152763 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace95297-0c31-45f3-8887-ce01eb29780f None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:41.177237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-ace95297-0c31-45f3-8887-ce01eb29780f None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 14:15:41.177237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:41.177237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:15:41.177237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:15:41.177237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:15:41.177237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:15:41.177237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:15:41.177237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:15:41.177237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 14:15:41.177237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:41.177237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:15:41.177237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:41.177237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:41.177237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:41.177237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:41.177237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:41.177237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:41.179224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:41.179224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:41.179224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:41.179224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:41.179224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:41.179224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:41.179224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:15:41.179224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:15:41.179224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:15:41.179224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:15:41.179224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:41.179224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:41.179224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:41.179224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:41.179224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:41.179224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:41.181615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:41.181615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:41.181615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:41.181615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:41.181615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:41.181615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:41.181615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:41.181615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:41.181615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:41.181615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:41.181615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:15:41.181615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:15:41.181615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 14:15:41.181615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:41.187712 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 353/702] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:40 2022] GET /identity/v3/services/network => generated 88 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:41.188984 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace95297-0c31-45f3-8887-ce01eb29780f None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:41.292659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ffe1d8-d649-461b-b01a-4253a049a822 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:41.363603 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a1f1a1-7ab1-4c4d-84c4-55a716cf8d4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:41.364241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a1f1a1-7ab1-4c4d-84c4-55a716cf8d4a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:41.364563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a1f1a1-7ab1-4c4d-84c4-55a716cf8d4a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:41.364833 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a1f1a1-7ab1-4c4d-84c4-55a716cf8d4a None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:41.365539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a1f1a1-7ab1-4c4d-84c4-55a716cf8d4a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:41.365845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a1f1a1-7ab1-4c4d-84c4-55a716cf8d4a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:41.367241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a1f1a1-7ab1-4c4d-84c4-55a716cf8d4a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:41.369453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace95297-0c31-45f3-8887-ce01eb29780f None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:41.378313 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 350/703] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:15:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:41.380430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a1f1a1-7ab1-4c4d-84c4-55a716cf8d4a None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:41.427160 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13985778-90bf-46f7-a533-2097a035db65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:41.427926 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13985778-90bf-46f7-a533-2097a035db65 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:41.428444 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13985778-90bf-46f7-a533-2097a035db65 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:41.428864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13985778-90bf-46f7-a533-2097a035db65 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:41.429670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13985778-90bf-46f7-a533-2097a035db65 None admin] RBAC: Adding query filter params (role.id=3ebe3c0be8c34c5eae533311e9b44176, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=5980e353164c4873bcdd2af847e0a1f1) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:41.430055 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13985778-90bf-46f7-a533-2097a035db65 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:41.431641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13985778-90bf-46f7-a533-2097a035db65 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:41.452647 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 354/704] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:15:41 2022] GET /identity/v3/role_assignments?user.id=5980e353164c4873bcdd2af847e0a1f1&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=3ebe3c0be8c34c5eae533311e9b44176 => generated 608 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:41.452647 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13985778-90bf-46f7-a533-2097a035db65 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:41.474903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:41.511667 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:41.567815 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a1f1a1-7ab1-4c4d-84c4-55a716cf8d4a None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:41.636206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3ffcaf-38a1-4f54-8adb-21a36d79f777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:41.636618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13985778-90bf-46f7-a533-2097a035db65 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:41.637097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3ffcaf-38a1-4f54-8adb-21a36d79f777 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:41.637542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3ffcaf-38a1-4f54-8adb-21a36d79f777 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:41.637975 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3ffcaf-38a1-4f54-8adb-21a36d79f777 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:41.639170 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3ffcaf-38a1-4f54-8adb-21a36d79f777 None admin] RBAC: Adding query filter params (name=network) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:41.639484 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3ffcaf-38a1-4f54-8adb-21a36d79f777 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:41.641861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3ffcaf-38a1-4f54-8adb-21a36d79f777 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:41.656899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 351/705] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Oct 4 14:15:41 2022] GET /identity/v3/services?name=network => generated 124 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:41.656899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde92d94-c37b-44bc-a866-085852facabf None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:41.656899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde92d94-c37b-44bc-a866-085852facabf None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:41.656899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde92d94-c37b-44bc-a866-085852facabf None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:41.656899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 352/706] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:41.656899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde92d94-c37b-44bc-a866-085852facabf None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:41.704832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca822f33-e69c-4c04-80a2-2062c2f608dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:41.705615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca822f33-e69c-4c04-80a2-2062c2f608dd None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:41.706000 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca822f33-e69c-4c04-80a2-2062c2f608dd None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:41.706381 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca822f33-e69c-4c04-80a2-2062c2f608dd None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:41.707145 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca822f33-e69c-4c04-80a2-2062c2f608dd None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:41.709280 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca822f33-e69c-4c04-80a2-2062c2f608dd None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:41.722740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-ca822f33-e69c-4c04-80a2-2062c2f608dd None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 14:15:41.722740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:41.722740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:15:41.722740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:15:41.722740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:15:41.722740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:15:41.722740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:15:41.722740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:15:41.722740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 14:15:41.722740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:41.722740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:15:41.722740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:41.722740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:41.722740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:41.722740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:41.722740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:41.722740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:41.724341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:41.724341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:41.724341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:41.724341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:41.724341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:41.724341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:41.724341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:15:41.724341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:15:41.724341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:15:41.724341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:15:41.724341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:41.724341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:41.724341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:41.724341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:41.724341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:41.724341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:41.725966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:41.725966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:41.725966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:41.725966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:41.725966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:41.725966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:41.725966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:41.725966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:41.725966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:41.725966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:41.725966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:15:41.725966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:15:41.725966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 14:15:41.725966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:41.729864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 355/707] 10.222.0.64 () {56 vars in 1129 bytes} [Tue Oct 4 14:15:41 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:41.730872 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca822f33-e69c-4c04-80a2-2062c2f608dd None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:41.834676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde92d94-c37b-44bc-a866-085852facabf None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:41.858507 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:41.892192 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62dd0d2-3f1e-46d4-8c72-2d597349b053 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:41.893551 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62dd0d2-3f1e-46d4-8c72-2d597349b053 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:41.893994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62dd0d2-3f1e-46d4-8c72-2d597349b053 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:41.894686 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62dd0d2-3f1e-46d4-8c72-2d597349b053 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:41.915389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62dd0d2-3f1e-46d4-8c72-2d597349b053 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:41.917616 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62dd0d2-3f1e-46d4-8c72-2d597349b053 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:41.921377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:41.927026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca822f33-e69c-4c04-80a2-2062c2f608dd None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:41.944872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-f62dd0d2-3f1e-46d4-8c72-2d597349b053 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:41.944872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:41.944872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:41.944872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:41.944872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:41.944872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:41.944872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:41.944872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:41.944872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:41.944872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:41.944872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:41.944872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:41.944872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:41.944872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:41.944872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:41.944872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:41.946823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:41.946823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:41.946823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:41.946823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:41.946823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:41.946823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:41.946823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:41.946823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:41.946823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:41.946823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:41.946823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:41.946823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:41.946823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:41.946823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:41.946823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:41.946823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:41.946823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:41.949343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:41.949343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:41.949343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:41.949343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 353/708] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:41.949343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c112f4f4-91e8-4695-906b-71afd501a146 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:41.949343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c112f4f4-91e8-4695-906b-71afd501a146 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:41.950606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c112f4f4-91e8-4695-906b-71afd501a146 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:41.974111 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-c112f4f4-91e8-4695-906b-71afd501a146 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:41.975542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c112f4f4-91e8-4695-906b-71afd501a146 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:41.998718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e01f97-e1db-4865-aa10-d94415d25847 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:41.999277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e01f97-e1db-4865-aa10-d94415d25847 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:41.999662 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e01f97-e1db-4865-aa10-d94415d25847 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:41.999797 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e01f97-e1db-4865-aa10-d94415d25847 None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:42.000443 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e01f97-e1db-4865-aa10-d94415d25847 None admin] RBAC: Adding query filter params (type=network) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:42.000753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e01f97-e1db-4865-aa10-d94415d25847 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:42.002176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e01f97-e1db-4865-aa10-d94415d25847 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:42.016373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 356/709] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Oct 4 14:15:41 2022] GET /identity/v3/services?type=network => generated 124 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:42.016373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e01f97-e1db-4865-aa10-d94415d25847 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:42.136580 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 354/710] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:42.138353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c112f4f4-91e8-4695-906b-71afd501a146 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:42.294443 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e01f97-e1db-4865-aa10-d94415d25847 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:42.366307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7cf0ff-a6ec-4cce-a491-6a1769d3e92e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:42.367374 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7cf0ff-a6ec-4cce-a491-6a1769d3e92e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:42.367825 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7cf0ff-a6ec-4cce-a491-6a1769d3e92e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:42.368277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7cf0ff-a6ec-4cce-a491-6a1769d3e92e None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:42.369471 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7cf0ff-a6ec-4cce-a491-6a1769d3e92e None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:42.369975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7cf0ff-a6ec-4cce-a491-6a1769d3e92e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:42.372488 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7cf0ff-a6ec-4cce-a491-6a1769d3e92e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:42.383111 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:42.392444 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 357/711] 10.222.0.64 () {56 vars in 1124 bytes} [Tue Oct 4 14:15:42 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 381 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:42.392711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c112f4f4-91e8-4695-906b-71afd501a146 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:42.401172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7cf0ff-a6ec-4cce-a491-6a1769d3e92e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:42.467508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:42.480741 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6325f975-596a-4ee2-86f6-9e2f21821558 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:42.482279 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6325f975-596a-4ee2-86f6-9e2f21821558 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:42.483132 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6325f975-596a-4ee2-86f6-9e2f21821558 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:42.483957 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6325f975-596a-4ee2-86f6-9e2f21821558 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:42.485379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6325f975-596a-4ee2-86f6-9e2f21821558 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:42.486659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6325f975-596a-4ee2-86f6-9e2f21821558 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:42.489574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6325f975-596a-4ee2-86f6-9e2f21821558 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:42.514288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 355/712] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 377 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:42.524648 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6325f975-596a-4ee2-86f6-9e2f21821558 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:42.633446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7cf0ff-a6ec-4cce-a491-6a1769d3e92e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:42.713981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12349db-c828-4204-bcf5-31f57ba51348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:42.715025 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12349db-c828-4204-bcf5-31f57ba51348 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:42.715582 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12349db-c828-4204-bcf5-31f57ba51348 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:42.716121 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12349db-c828-4204-bcf5-31f57ba51348 None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:42.718139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12349db-c828-4204-bcf5-31f57ba51348 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:42.718833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12349db-c828-4204-bcf5-31f57ba51348 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:42.722007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12349db-c828-4204-bcf5-31f57ba51348 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:42.732440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 358/713] 10.222.0.64 () {56 vars in 1124 bytes} [Tue Oct 4 14:15:42 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:15:42.734946 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12349db-c828-4204-bcf5-31f57ba51348 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:42.781987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6325f975-596a-4ee2-86f6-9e2f21821558 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:42.860747 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1780c86-a444-4bf2-b0f0-69441f56bb58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:42.861900 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1780c86-a444-4bf2-b0f0-69441f56bb58 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:42.862446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1780c86-a444-4bf2-b0f0-69441f56bb58 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:42.862996 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1780c86-a444-4bf2-b0f0-69441f56bb58 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:42.878956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1780c86-a444-4bf2-b0f0-69441f56bb58 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:42.881730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1780c86-a444-4bf2-b0f0-69441f56bb58 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:42.908381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-e1780c86-a444-4bf2-b0f0-69441f56bb58 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:15:42.908381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:42.908381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:42.908381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:42.908381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:42.908381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:42.908381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:42.908381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:42.908381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:42.908381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:42.908381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:42.908381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:42.908381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:42.908381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:42.908381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:42.908381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:42.910244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:42.912860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:42.912860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:42.912860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:42.912860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:42.912860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:42.912860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:42.912860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:42.912860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:42.912860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:15:42.912860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:42.916751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 356/714] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Oct 4 14:15:42 2022] GET /identity/v3/users/nova => generated 82 bytes in 387 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:42.922963 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1780c86-a444-4bf2-b0f0-69441f56bb58 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:42.986953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:43.051402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:43.087062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12349db-c828-4204-bcf5-31f57ba51348 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:43.176395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1780c86-a444-4bf2-b0f0-69441f56bb58 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:43.194158 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4728595a-df80-4b02-864f-711bd348f938 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:43.195751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4728595a-df80-4b02-864f-711bd348f938 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:43.196540 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4728595a-df80-4b02-864f-711bd348f938 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:43.197416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4728595a-df80-4b02-864f-711bd348f938 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:43.224196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4728595a-df80-4b02-864f-711bd348f938 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:43.235991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-4728595a-df80-4b02-864f-711bd348f938 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:43.235991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:43.235991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:43.235991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:43.235991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:43.235991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:43.235991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:43.235991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:43.235991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:43.235991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:43.235991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:43.235991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:43.235991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:43.235991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:43.235991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:43.235991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:43.237867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:43.237867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:43.237867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:43.237867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:43.237867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:43.237867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:43.237867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:43.237867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:43.237867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:43.237867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:43.237867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:43.237867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:43.237867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:43.237867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:43.237867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:43.237867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:43.239686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:43.239686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:43.239686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:43.239686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:43.239686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:43.239686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:43.239686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:43.239686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:43.239686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:43.239686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:43.239686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:43.239686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:43.243130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697d14f0-b2fc-4e71-b6d7-93eec0a6a33a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:43.244124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697d14f0-b2fc-4e71-b6d7-93eec0a6a33a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:43.244639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697d14f0-b2fc-4e71-b6d7-93eec0a6a33a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:43.245105 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697d14f0-b2fc-4e71-b6d7-93eec0a6a33a None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:43.246014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 359/715] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:42 2022] GET /identity/v3/roles/service => generated 85 bytes in 512 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:43.246503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697d14f0-b2fc-4e71-b6d7-93eec0a6a33a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:43.246950 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697d14f0-b2fc-4e71-b6d7-93eec0a6a33a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:43.249944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697d14f0-b2fc-4e71-b6d7-93eec0a6a33a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:43.256623 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4728595a-df80-4b02-864f-711bd348f938 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:43.288487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 357/716] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Oct 4 14:15:42 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 364 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:43.293111 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697d14f0-b2fc-4e71-b6d7-93eec0a6a33a None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:43.445404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:43.488621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697d14f0-b2fc-4e71-b6d7-93eec0a6a33a None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:43.510474 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4728595a-df80-4b02-864f-711bd348f938 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:43.563599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28572bc-cb22-4bbb-817c-907dd0b53e44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:43.564647 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28572bc-cb22-4bbb-817c-907dd0b53e44 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:43.565111 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28572bc-cb22-4bbb-817c-907dd0b53e44 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:43.565713 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28572bc-cb22-4bbb-817c-907dd0b53e44 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:43.585243 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e9304f-80a1-4394-8c3f-ed7936362f4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:43.586970 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e9304f-80a1-4394-8c3f-ed7936362f4b None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:43.587794 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e9304f-80a1-4394-8c3f-ed7936362f4b None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:43.588591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e9304f-80a1-4394-8c3f-ed7936362f4b None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:43.590161 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e9304f-80a1-4394-8c3f-ed7936362f4b None admin] RBAC: Adding query filter params (name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:43.591024 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e9304f-80a1-4394-8c3f-ed7936362f4b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:43.594258 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e9304f-80a1-4394-8c3f-ed7936362f4b None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:43.615185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 360/717] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:15:43 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 360 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:43.623600 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e9304f-80a1-4394-8c3f-ed7936362f4b None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:43.669366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:43.788054 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28572bc-cb22-4bbb-817c-907dd0b53e44 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:43.789507 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28572bc-cb22-4bbb-817c-907dd0b53e44 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:43.987711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e9304f-80a1-4394-8c3f-ed7936362f4b None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:43.999490 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:44.057463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 358/718] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:15:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 765 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 14:15:44.066456 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28572bc-cb22-4bbb-817c-907dd0b53e44 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:44.080551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4375882-a916-4d83-8a74-872f6a3d8f6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:44.081777 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4375882-a916-4d83-8a74-872f6a3d8f6e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:44.082344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4375882-a916-4d83-8a74-872f6a3d8f6e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:44.082800 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4375882-a916-4d83-8a74-872f6a3d8f6e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:44.097901 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4375882-a916-4d83-8a74-872f6a3d8f6e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:44.099459 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4375882-a916-4d83-8a74-872f6a3d8f6e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:44.117363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-b4375882-a916-4d83-8a74-872f6a3d8f6e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:44.117363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:44.117363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:44.117363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:44.117363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:44.117363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:44.117363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:44.117363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:44.117363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:44.117363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:44.117363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:44.117363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:44.117363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:44.117363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:44.117363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:44.117363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:44.119048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:44.119048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:44.119048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:44.119048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:44.119048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:44.119048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:44.119048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:44.119048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:44.119048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:44.119048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:44.119048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:44.119048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:44.119048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:44.119048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:44.119048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:44.119048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:44.119048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:44.121422 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:44.121422 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:44.121422 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:44.121422 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 361/719] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 497 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:44.126050 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4375882-a916-4d83-8a74-872f6a3d8f6e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:44.288862 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:44.375446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28572bc-cb22-4bbb-817c-907dd0b53e44 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:44.443317 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c387d7f-5e15-463b-93a8-863b6dba9082 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:44.444408 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c387d7f-5e15-463b-93a8-863b6dba9082 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:44.444886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c387d7f-5e15-463b-93a8-863b6dba9082 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:44.445314 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c387d7f-5e15-463b-93a8-863b6dba9082 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:44.466459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c387d7f-5e15-463b-93a8-863b6dba9082 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:44.469815 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c387d7f-5e15-463b-93a8-863b6dba9082 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:44.471685 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4375882-a916-4d83-8a74-872f6a3d8f6e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:44.499930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-7c387d7f-5e15-463b-93a8-863b6dba9082 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:44.499930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:44.499930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:44.499930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:44.499930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:44.499930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:44.499930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:44.499930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:44.499930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:44.499930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:44.499930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:44.499930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:44.499930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:44.499930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:44.499930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:44.499930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:44.501187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:44.501187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:44.501187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:44.501187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:44.501187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:44.501187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:44.501187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:44.501187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:44.501187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:44.501187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:44.501187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:44.501187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:44.501187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:44.501187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:44.501187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:44.501187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:44.501187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:44.502602 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:44.502602 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:44.502602 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:44.502602 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 359/720] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 437 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:44.508952 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c387d7f-5e15-463b-93a8-863b6dba9082 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:44.574437 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:44.609515 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3857b0-e621-4b2b-83e5-217e0eced06d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:44.610165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3857b0-e621-4b2b-83e5-217e0eced06d None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:44.610452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3857b0-e621-4b2b-83e5-217e0eced06d None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:44.610732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3857b0-e621-4b2b-83e5-217e0eced06d None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:44.611406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3857b0-e621-4b2b-83e5-217e0eced06d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:44.611714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3857b0-e621-4b2b-83e5-217e0eced06d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:44.613175 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3857b0-e621-4b2b-83e5-217e0eced06d None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:44.643112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 362/721] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 517 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:44.663735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3857b0-e621-4b2b-83e5-217e0eced06d None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:44.745580 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c387d7f-5e15-463b-93a8-863b6dba9082 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:44.819127 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6a9f64-4d13-4e6d-9510-83f5772a52c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:44.820491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6a9f64-4d13-4e6d-9510-83f5772a52c1 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:44.821667 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6a9f64-4d13-4e6d-9510-83f5772a52c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:44.822525 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6a9f64-4d13-4e6d-9510-83f5772a52c1 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:44.823695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6a9f64-4d13-4e6d-9510-83f5772a52c1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:44.824212 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6a9f64-4d13-4e6d-9510-83f5772a52c1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:44.826768 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6a9f64-4d13-4e6d-9510-83f5772a52c1 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:44.860362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 360/722] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 349 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:44.860362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d01047e-68d5-4832-a5a4-7018aaf7b701 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:44.860362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d01047e-68d5-4832-a5a4-7018aaf7b701 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:44.860362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d01047e-68d5-4832-a5a4-7018aaf7b701 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:44.861983 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 361/723] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:44 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:44.863986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c533822e-a5cc-424a-b7ef-a6c09c833397 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:44.864530 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c533822e-a5cc-424a-b7ef-a6c09c833397 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:44.867002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c533822e-a5cc-424a-b7ef-a6c09c833397 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:44.867420 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 362/724] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:44 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:44.868769 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c533822e-a5cc-424a-b7ef-a6c09c833397 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:44.907093 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3857b0-e621-4b2b-83e5-217e0eced06d None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:44.959944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:44.974268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0b5d35-9d90-461a-aaf6-c8628dc1b430 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:44.975115 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0b5d35-9d90-461a-aaf6-c8628dc1b430 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:44.975515 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0b5d35-9d90-461a-aaf6-c8628dc1b430 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:44.975937 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0b5d35-9d90-461a-aaf6-c8628dc1b430 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:44.990207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0b5d35-9d90-461a-aaf6-c8628dc1b430 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:44.991910 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0b5d35-9d90-461a-aaf6-c8628dc1b430 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:45.005406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-2b0b5d35-9d90-461a-aaf6-c8628dc1b430 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 14:15:45.005406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:45.005406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:45.005406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:45.005406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:45.005406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:45.005406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:45.005406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:45.005406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:45.005406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:45.005406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:45.005406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:45.005406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:45.005406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:45.005406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:45.005406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:45.007331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:45.009988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:45.009988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:45.009988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:45.009988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:45.009988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:45.009988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:45.009988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:45.009988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:45.009988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 14:15:45.009988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:45.009988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 363/725] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Oct 4 14:15:44 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 344 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:45.011136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d396880-c532-4d7c-af3c-79da35162a5b None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:45.011136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d396880-c532-4d7c-af3c-79da35162a5b None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:45.011374 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d396880-c532-4d7c-af3c-79da35162a5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:45.031416 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:45.041867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-5d396880-c532-4d7c-af3c-79da35162a5b None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:45.044556 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-5d396880-c532-4d7c-af3c-79da35162a5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:45.046695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c533822e-a5cc-424a-b7ef-a6c09c833397 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:45.099228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d16ec4-a77f-4e2c-ae3c-7b7e549441da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:45.099966 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d16ec4-a77f-4e2c-ae3c-7b7e549441da None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:45.100246 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d16ec4-a77f-4e2c-ae3c-7b7e549441da None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:45.100620 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d16ec4-a77f-4e2c-ae3c-7b7e549441da None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:45.110531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d16ec4-a77f-4e2c-ae3c-7b7e549441da None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:45.112290 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d16ec4-a77f-4e2c-ae3c-7b7e549441da None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:45.126004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-02d16ec4-a77f-4e2c-ae3c-7b7e549441da None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:45.126004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:45.126004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:45.126004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:45.126004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:45.126004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:45.126004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:45.126004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:45.126004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:45.126004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:45.126004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:45.126004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:45.126004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:45.126004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:45.126004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:45.127889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:45.127889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:45.127889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:45.127889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:45.127889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:45.127889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:45.127889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:45.127889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:45.127889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:45.127889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:45.127889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:45.127889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:45.127889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:45.127889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:45.127889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:45.127889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:45.127889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:45.129822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:45.129822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:45.129822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:45.129822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:45.129822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:45.129822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:45.129822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:45.129822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:45.129822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 363/726] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:45.129822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29175f1a-f9bb-47ed-a929-c8e0009a8a80 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:45.129822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29175f1a-f9bb-47ed-a929-c8e0009a8a80 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:45.131008 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29175f1a-f9bb-47ed-a929-c8e0009a8a80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:45.152484 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-29175f1a-f9bb-47ed-a929-c8e0009a8a80 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:45.153508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-29175f1a-f9bb-47ed-a929-c8e0009a8a80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:45.187060 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 364/727] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:45.187918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d396880-c532-4d7c-af3c-79da35162a5b None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:45.297885 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 364/728] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:45.299106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29175f1a-f9bb-47ed-a929-c8e0009a8a80 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:45.477185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d396880-c532-4d7c-af3c-79da35162a5b None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:45.535986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:45.545001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:45.578526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29175f1a-f9bb-47ed-a929-c8e0009a8a80 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:45.613566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e5eebb-752f-4e78-a61b-c285530234ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:45.614521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e5eebb-752f-4e78-a61b-c285530234ce None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:45.614989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e5eebb-752f-4e78-a61b-c285530234ce None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:45.615456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e5eebb-752f-4e78-a61b-c285530234ce None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:45.616788 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e5eebb-752f-4e78-a61b-c285530234ce None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:45.617316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e5eebb-752f-4e78-a61b-c285530234ce None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:45.619871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e5eebb-752f-4e78-a61b-c285530234ce None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:45.657463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3b0a8b-19a9-4d7b-8925-c204e264c139 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:45.658982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3b0a8b-19a9-4d7b-8925-c204e264c139 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:45.659768 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3b0a8b-19a9-4d7b-8925-c204e264c139 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:45.660521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3b0a8b-19a9-4d7b-8925-c204e264c139 None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:45.662102 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3b0a8b-19a9-4d7b-8925-c204e264c139 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:45.662891 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3b0a8b-19a9-4d7b-8925-c204e264c139 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:45.670621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3b0a8b-19a9-4d7b-8925-c204e264c139 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:45.682572 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 365/729] 10.222.0.64 () {56 vars in 1149 bytes} [Tue Oct 4 14:15:45 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 495 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:45.692228 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e5eebb-752f-4e78-a61b-c285530234ce None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:45.697966 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 365/730] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 400 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:45.707560 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3b0a8b-19a9-4d7b-8925-c204e264c139 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:45.907568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e5eebb-752f-4e78-a61b-c285530234ce None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:45.913423 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3b0a8b-19a9-4d7b-8925-c204e264c139 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:45.969838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20548d86-8b47-401c-ae82-004141856677 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:45.970920 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20548d86-8b47-401c-ae82-004141856677 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:45.971377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20548d86-8b47-401c-ae82-004141856677 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:45.971820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20548d86-8b47-401c-ae82-004141856677 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:45.974673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8ee519-fcff-426b-b331-e1aa0db49c52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:45.975308 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8ee519-fcff-426b-b331-e1aa0db49c52 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:45.975576 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8ee519-fcff-426b-b331-e1aa0db49c52 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:45.975849 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8ee519-fcff-426b-b331-e1aa0db49c52 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:45.976615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8ee519-fcff-426b-b331-e1aa0db49c52 None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=03081d7d581745c88c12e9d089fcb2d3) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:45.976924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8ee519-fcff-426b-b331-e1aa0db49c52 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:45.978404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8ee519-fcff-426b-b331-e1aa0db49c52 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:45.985673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 366/731] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:15:45 2022] GET /identity/v3/role_assignments?user.id=03081d7d581745c88c12e9d089fcb2d3&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 259 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:45.986523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8ee519-fcff-426b-b331-e1aa0db49c52 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:45.998598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:46.073002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:46.135882 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20548d86-8b47-401c-ae82-004141856677 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:46.137839 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20548d86-8b47-401c-ae82-004141856677 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:46.206989 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8ee519-fcff-426b-b331-e1aa0db49c52 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:46.276779 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67647012-0f17-463d-aa03-585cd84b69a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:46.277871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67647012-0f17-463d-aa03-585cd84b69a0 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:46.278388 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67647012-0f17-463d-aa03-585cd84b69a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:46.278896 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67647012-0f17-463d-aa03-585cd84b69a0 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:46.280943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67647012-0f17-463d-aa03-585cd84b69a0 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:46.283512 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67647012-0f17-463d-aa03-585cd84b69a0 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:46.307956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 367/732] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Oct 4 14:15:45 2022] POST /identity/v3/services => generated 240 bytes in 322 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:15:46.309907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67647012-0f17-463d-aa03-585cd84b69a0 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:46.329753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 366/733] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:15:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 638 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 14:15:46.334234 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78401aa6-8986-4f84-a74d-0cf6b83b6464 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:46.335828 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78401aa6-8986-4f84-a74d-0cf6b83b6464 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:46.336466 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78401aa6-8986-4f84-a74d-0cf6b83b6464 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:46.343412 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 367/734] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:46 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:46.344987 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78401aa6-8986-4f84-a74d-0cf6b83b6464 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:46.461465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:46.463448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:46.516220 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78401aa6-8986-4f84-a74d-0cf6b83b6464 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:46.542937 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67647012-0f17-463d-aa03-585cd84b69a0 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:46.562719 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18624d5d-317c-491b-81e2-497050c931ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:46.563817 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18624d5d-317c-491b-81e2-497050c931ca None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:46.564442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18624d5d-317c-491b-81e2-497050c931ca None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:46.564955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18624d5d-317c-491b-81e2-497050c931ca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:46.579824 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18624d5d-317c-491b-81e2-497050c931ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:46.581702 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18624d5d-317c-491b-81e2-497050c931ca None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:46.600627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-18624d5d-317c-491b-81e2-497050c931ca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:46.600627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:46.600627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:46.600627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:46.600627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:46.600627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:46.600627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:46.600627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:46.600627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:46.600627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:46.600627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:46.600627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:46.600627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:46.600627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:46.600627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:46.600627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:46.601654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:46.601654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:46.601654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:46.601654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:46.601654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:46.601654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:46.601654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:46.601654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:46.601654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:46.601654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:46.601654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:46.601654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:46.601654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:46.601654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:46.601654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:46.601654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:46.601654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:46.602566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:46.602566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:46.602566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:46.604927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 368/735] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:46.606853 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968c30af-140e-49f3-9987-b80925f0dab5 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:46.607404 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968c30af-140e-49f3-9987-b80925f0dab5 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:46.607855 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968c30af-140e-49f3-9987-b80925f0dab5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:46.609078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ea51bf-63b5-4226-b8e5-6d4344c1fea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:46.610226 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ea51bf-63b5-4226-b8e5-6d4344c1fea1 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:46.610670 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ea51bf-63b5-4226-b8e5-6d4344c1fea1 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:46.611009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ea51bf-63b5-4226-b8e5-6d4344c1fea1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:46.633263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-968c30af-140e-49f3-9987-b80925f0dab5 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:46.634807 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-968c30af-140e-49f3-9987-b80925f0dab5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:46.635685 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ea51bf-63b5-4226-b8e5-6d4344c1fea1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:46.650737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-70ea51bf-63b5-4226-b8e5-6d4344c1fea1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:15:46.650737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:46.650737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:46.650737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:46.650737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:46.650737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:46.650737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:46.650737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:46.650737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:46.650737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:46.650737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:46.650737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:46.650737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:46.650737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:46.650737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:46.650737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:46.652047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:46.652047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:46.652047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:46.652047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:46.652047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:46.652047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:46.652047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:46.652047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:46.652047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:46.652047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:46.652047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:46.652047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:46.652047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:46.652047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:46.652047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:46.652047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:46.653253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:46.653253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:46.653253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:46.653253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:46.653253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:46.653253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:46.653253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:46.653253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:46.653253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:46.653253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:46.653253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:15:46.653253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:46.653253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 368/736] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:15:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 344 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:46.654723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ea51bf-63b5-4226-b8e5-6d4344c1fea1 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:46.780424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 369/737] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:46 2022] POST /identity/v3/auth/tokens => generated 898 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 14:15:46.780424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968c30af-140e-49f3-9987-b80925f0dab5 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:46.823441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ea51bf-63b5-4226-b8e5-6d4344c1fea1 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:46.852333 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:46.866632 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee3a13d-6682-4a08-9a06-7deaf9961a7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:46.867255 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee3a13d-6682-4a08-9a06-7deaf9961a7e None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:46.867537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee3a13d-6682-4a08-9a06-7deaf9961a7e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:46.867811 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee3a13d-6682-4a08-9a06-7deaf9961a7e None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:46.868590 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee3a13d-6682-4a08-9a06-7deaf9961a7e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:46.868899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee3a13d-6682-4a08-9a06-7deaf9961a7e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:46.870310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee3a13d-6682-4a08-9a06-7deaf9961a7e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:46.892346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 369/738] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:46.895054 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee3a13d-6682-4a08-9a06-7deaf9961a7e None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:46.928279 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:47.040640 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee3a13d-6682-4a08-9a06-7deaf9961a7e None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:47.085599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262f3d47-f5d9-4190-b044-23694d235896 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:47.086121 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262f3d47-f5d9-4190-b044-23694d235896 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:47.086398 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262f3d47-f5d9-4190-b044-23694d235896 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:47.086667 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262f3d47-f5d9-4190-b044-23694d235896 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:47.100183 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968c30af-140e-49f3-9987-b80925f0dab5 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:47.103697 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-262f3d47-f5d9-4190-b044-23694d235896 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:47.105221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-262f3d47-f5d9-4190-b044-23694d235896 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:47.126548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-262f3d47-f5d9-4190-b044-23694d235896 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:47.126548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:47.126548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:47.126548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:47.126548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:47.126548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:47.126548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:47.126548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:47.126548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:47.126548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:47.126548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:47.126548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:47.126548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:47.126548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:47.126548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:47.127550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:47.127550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:47.127550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:47.127550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:47.127550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:47.127550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:47.127550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:47.127550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:47.127550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:47.127550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:47.127550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:47.127550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:47.127550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:47.127550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:47.127550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:47.127550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:47.127550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:47.128889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:47.128889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:47.128889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:47.128889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:47.128889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:47.128889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:47.128889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:47.128889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:47.128889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 370/739] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:47.128889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262f3d47-f5d9-4190-b044-23694d235896 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:47.161189 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c937e7d4-0312-481c-ae6b-08acf825682c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:47.163149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c937e7d4-0312-481c-ae6b-08acf825682c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:47.163645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c937e7d4-0312-481c-ae6b-08acf825682c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:47.164074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c937e7d4-0312-481c-ae6b-08acf825682c None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:47.165191 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c937e7d4-0312-481c-ae6b-08acf825682c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:47.166182 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c937e7d4-0312-481c-ae6b-08acf825682c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:47.168146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c937e7d4-0312-481c-ae6b-08acf825682c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:47.181742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 370/740] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:15:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 407 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:47.182961 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c937e7d4-0312-481c-ae6b-08acf825682c None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:47.189618 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:47.255029 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262f3d47-f5d9-4190-b044-23694d235896 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:47.294653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a367772-8338-4b00-88c0-6f3ff1e993ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:47.295384 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a367772-8338-4b00-88c0-6f3ff1e993ca None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:47.296303 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a367772-8338-4b00-88c0-6f3ff1e993ca None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:47.297413 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a367772-8338-4b00-88c0-6f3ff1e993ca None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:47.298106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a367772-8338-4b00-88c0-6f3ff1e993ca None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:47.299516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a367772-8338-4b00-88c0-6f3ff1e993ca None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:47.307943 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c937e7d4-0312-481c-ae6b-08acf825682c None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:47.326332 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 371/741] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Oct 4 14:15:47 2022] POST /identity/v3/services => generated 251 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:15:47.326332 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a367772-8338-4b00-88c0-6f3ff1e993ca None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:47.332296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:47.376651 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757f0ddd-6f5f-4fc8-ae50-0afd8d7bdedd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:47.377452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757f0ddd-6f5f-4fc8-ae50-0afd8d7bdedd None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:47.377832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757f0ddd-6f5f-4fc8-ae50-0afd8d7bdedd None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:47.378210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757f0ddd-6f5f-4fc8-ae50-0afd8d7bdedd None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:47.379701 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757f0ddd-6f5f-4fc8-ae50-0afd8d7bdedd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:47.380568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757f0ddd-6f5f-4fc8-ae50-0afd8d7bdedd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:47.382428 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757f0ddd-6f5f-4fc8-ae50-0afd8d7bdedd None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:47.417284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 371/742] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:47.425653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757f0ddd-6f5f-4fc8-ae50-0afd8d7bdedd None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:47.575394 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a367772-8338-4b00-88c0-6f3ff1e993ca None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:47.579469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:47.619650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757f0ddd-6f5f-4fc8-ae50-0afd8d7bdedd None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:47.648199 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7986ffe-e40d-4146-8045-4ab3a78513f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:47.649079 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7986ffe-e40d-4146-8045-4ab3a78513f0 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:47.649435 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7986ffe-e40d-4146-8045-4ab3a78513f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:47.649710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7986ffe-e40d-4146-8045-4ab3a78513f0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:47.666092 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7986ffe-e40d-4146-8045-4ab3a78513f0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:47.666499 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd444465-adfa-4936-ace7-e3e6ab8fe054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:47.667072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd444465-adfa-4936-ace7-e3e6ab8fe054 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:47.667353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd444465-adfa-4936-ace7-e3e6ab8fe054 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:47.667627 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7986ffe-e40d-4146-8045-4ab3a78513f0 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:47.667778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd444465-adfa-4936-ace7-e3e6ab8fe054 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:47.668467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd444465-adfa-4936-ace7-e3e6ab8fe054 None admin] RBAC: Adding query filter params (role.id=3ebe3c0be8c34c5eae533311e9b44176, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=34afa4aac10c4a018ca05a0ba34c940a) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:47.668772 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd444465-adfa-4936-ace7-e3e6ab8fe054 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:47.670344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd444465-adfa-4936-ace7-e3e6ab8fe054 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:47.683097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 372/743] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:15:47 2022] GET /identity/v3/role_assignments?user.id=34afa4aac10c4a018ca05a0ba34c940a&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=3ebe3c0be8c34c5eae533311e9b44176 => generated 259 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:15:47.701287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-e7986ffe-e40d-4146-8045-4ab3a78513f0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:47.701287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:47.701287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:47.701287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:47.701287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:47.701287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:47.701287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:47.701287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:47.701287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:47.701287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:47.701287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:47.701287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:47.701287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:47.701287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:47.701287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:47.701287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:47.703882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:47.703882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:47.703882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:47.703882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:47.703882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:47.703882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:47.703882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:47.703882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:47.703882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:47.703882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:47.703882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:47.703882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:47.703882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:47.703882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:47.703882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:47.703882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:47.703882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:47.707404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:47.707404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:47.707404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:47.711078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 372/744] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 386 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:47.716551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd444465-adfa-4936-ace7-e3e6ab8fe054 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:47.809994 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:47.959601 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd444465-adfa-4936-ace7-e3e6ab8fe054 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:48.027906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e8cf3e-fa6e-4d12-8813-1399b7e6c0ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:48.029023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e8cf3e-fa6e-4d12-8813-1399b7e6c0ad None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:48.029758 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e8cf3e-fa6e-4d12-8813-1399b7e6c0ad None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:48.030898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e8cf3e-fa6e-4d12-8813-1399b7e6c0ad None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:48.032053 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e8cf3e-fa6e-4d12-8813-1399b7e6c0ad None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:48.032583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e8cf3e-fa6e-4d12-8813-1399b7e6c0ad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:48.035211 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e8cf3e-fa6e-4d12-8813-1399b7e6c0ad None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:48.059853 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 373/745] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 344 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:48.066089 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7986ffe-e40d-4146-8045-4ab3a78513f0 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:48.271921 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7986ffe-e40d-4146-8045-4ab3a78513f0 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:48.328569 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648b647c-5dea-4b4e-ad4b-0be8800c7fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:48.329222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648b647c-5dea-4b4e-ad4b-0be8800c7fba None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:48.329498 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648b647c-5dea-4b4e-ad4b-0be8800c7fba None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:48.329764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648b647c-5dea-4b4e-ad4b-0be8800c7fba None admin] PATH_INFO: `/v3/users/swift` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:48.339557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648b647c-5dea-4b4e-ad4b-0be8800c7fba None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:48.341083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648b647c-5dea-4b4e-ad4b-0be8800c7fba None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:48.357733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-648b647c-5dea-4b4e-ad4b-0be8800c7fba None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 14:15:48.357733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:48.357733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:48.357733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:48.357733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:48.357733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:48.357733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:48.357733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:48.357733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:48.357733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:48.357733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:48.357733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:48.357733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:48.357733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:48.357733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:48.357733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:48.360478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:48.362753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:48.362753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:48.362753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:48.362753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:48.362753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:48.362753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:48.362753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:48.362753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:48.362753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 14:15:48.362753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:48.362753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 373/746] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:15:48 2022] GET /identity/v3/users/swift => generated 83 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:48.369006 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e8cf3e-fa6e-4d12-8813-1399b7e6c0ad None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:48.628420 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:48.640289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e8cf3e-fa6e-4d12-8813-1399b7e6c0ad None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:48.709339 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09148879-1e4b-4fa6-bb00-d0bbb3e2c20b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:48.710383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09148879-1e4b-4fa6-bb00-d0bbb3e2c20b None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:48.710960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09148879-1e4b-4fa6-bb00-d0bbb3e2c20b None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:48.711452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09148879-1e4b-4fa6-bb00-d0bbb3e2c20b None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:48.712972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09148879-1e4b-4fa6-bb00-d0bbb3e2c20b None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:48.713505 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09148879-1e4b-4fa6-bb00-d0bbb3e2c20b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:48.716430 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09148879-1e4b-4fa6-bb00-d0bbb3e2c20b None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:48.755836 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 374/747] 10.222.0.64 () {56 vars in 1135 bytes} [Tue Oct 4 14:15:48 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 388 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:15:48.761946 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648b647c-5dea-4b4e-ad4b-0be8800c7fba None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:48.803567 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:48.948878 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648b647c-5dea-4b4e-ad4b-0be8800c7fba None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:49.007709 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4329e0c-9f45-4eb4-b02b-65abb7970ab6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:49.008926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4329e0c-9f45-4eb4-b02b-65abb7970ab6 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:49.009447 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4329e0c-9f45-4eb4-b02b-65abb7970ab6 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:49.010017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4329e0c-9f45-4eb4-b02b-65abb7970ab6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:49.194227 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4329e0c-9f45-4eb4-b02b-65abb7970ab6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:49.197023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4329e0c-9f45-4eb4-b02b-65abb7970ab6 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:49.246467 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:49.389372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 374/748] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:15:48 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 628 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 14:15:49.396347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09148879-1e4b-4fa6-bb00-d0bbb3e2c20b None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:49.443846 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e18354-f1c4-49c4-8b06-23f47d13805a None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:49.444236 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e18354-f1c4-49c4-8b06-23f47d13805a None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:49.444444 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e18354-f1c4-49c4-8b06-23f47d13805a None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:49.446685 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 375/749] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:49.454005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841b6059-3f13-4d9e-ac22-40f70bec2ad7 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:49.454317 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841b6059-3f13-4d9e-ac22-40f70bec2ad7 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:49.454540 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841b6059-3f13-4d9e-ac22-40f70bec2ad7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:49.477062 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-841b6059-3f13-4d9e-ac22-40f70bec2ad7 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:49.478990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-841b6059-3f13-4d9e-ac22-40f70bec2ad7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:49.561800 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 376/750] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:49 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:15:49.568483 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dbcb1e-91b9-4b85-b600-b36059b4d999 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:49.569145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dbcb1e-91b9-4b85-b600-b36059b4d999 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:49.569627 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dbcb1e-91b9-4b85-b600-b36059b4d999 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:49.571537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 377/751] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:15:49.578974 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86df05e-206c-48d6-8df3-3af68d69de84 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:49.579649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86df05e-206c-48d6-8df3-3af68d69de84 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:49.580150 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86df05e-206c-48d6-8df3-3af68d69de84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:49.624612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-c86df05e-206c-48d6-8df3-3af68d69de84 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:49.626830 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c86df05e-206c-48d6-8df3-3af68d69de84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:49.730132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09148879-1e4b-4fa6-bb00-d0bbb3e2c20b None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:49.732290 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:49.786599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 378/752] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:49 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:15:49.802008 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5dca866-b928-4ec6-879a-1c87aaff89b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:49.802841 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5dca866-b928-4ec6-879a-1c87aaff89b8 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:49.803221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5dca866-b928-4ec6-879a-1c87aaff89b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:49.803612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5dca866-b928-4ec6-879a-1c87aaff89b8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:49.825419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5dca866-b928-4ec6-879a-1c87aaff89b8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:49.828034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5dca866-b928-4ec6-879a-1c87aaff89b8 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:49.854176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-d5dca866-b928-4ec6-879a-1c87aaff89b8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:49.854176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:49.854176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:49.854176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:49.854176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:49.854176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:49.854176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:49.854176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:49.854176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:49.854176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:49.854176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:49.854176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:49.854176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:49.854176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:49.854176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:49.854176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:49.856302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:49.856302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:49.856302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:49.856302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:49.856302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:49.856302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:49.856302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:49.856302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:49.856302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:49.856302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:49.856302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:49.856302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:49.856302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:49.856302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:49.856302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:49.856302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:49.856302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:49.858450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:49.858450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:49.858450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:49.858450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 375/753] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 460 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:49.866160 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86df05e-206c-48d6-8df3-3af68d69de84 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:49.894348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5dca866-b928-4ec6-879a-1c87aaff89b8 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:49.900068 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:50.046768 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86df05e-206c-48d6-8df3-3af68d69de84 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:50.091075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b38f2b-4358-4b98-92ce-2b5a74be9af5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:50.091595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b38f2b-4358-4b98-92ce-2b5a74be9af5 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:50.091865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b38f2b-4358-4b98-92ce-2b5a74be9af5 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:50.092133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b38f2b-4358-4b98-92ce-2b5a74be9af5 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:50.093631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b38f2b-4358-4b98-92ce-2b5a74be9af5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:50.094014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b38f2b-4358-4b98-92ce-2b5a74be9af5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:50.095542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b38f2b-4358-4b98-92ce-2b5a74be9af5 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:50.111634 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 379/754] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:50.113894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5dca866-b928-4ec6-879a-1c87aaff89b8 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:50.127472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b38f2b-4358-4b98-92ce-2b5a74be9af5 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:50.162041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a63466e-6eff-4042-98e0-4e0838ee908b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:50.162737 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a63466e-6eff-4042-98e0-4e0838ee908b None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:50.163014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a63466e-6eff-4042-98e0-4e0838ee908b None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:50.163298 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a63466e-6eff-4042-98e0-4e0838ee908b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:50.178604 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a63466e-6eff-4042-98e0-4e0838ee908b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:50.184473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:50.188173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-0a63466e-6eff-4042-98e0-4e0838ee908b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:15:50.188173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:50.188173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:50.188173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:50.188173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:50.188173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:50.188173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:50.188173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:50.188173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:50.188173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:50.188173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:50.188173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:50.188173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:50.188173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:50.188173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:50.188173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:50.189222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:50.189222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:50.189222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:50.189222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:50.189222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:50.189222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:50.189222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:50.189222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:50.189222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:50.189222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:50.189222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:50.189222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:50.189222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:50.189222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:50.189222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:50.189222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:50.190073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:50.190073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:50.190073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:50.190073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:50.190073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:50.190073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:50.190073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:50.190073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:50.190073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:50.190073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:50.190073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:15:50.190073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:50.190748 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 376/755] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:15:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:50.200526 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a63466e-6eff-4042-98e0-4e0838ee908b None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:50.293980 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b38f2b-4358-4b98-92ce-2b5a74be9af5 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:50.323268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:50.338025 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2b355f-a8ec-45a8-a3b9-22bf3078cc02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:50.339104 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2b355f-a8ec-45a8-a3b9-22bf3078cc02 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:50.339643 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2b355f-a8ec-45a8-a3b9-22bf3078cc02 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:50.340119 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2b355f-a8ec-45a8-a3b9-22bf3078cc02 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:50.350481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2b355f-a8ec-45a8-a3b9-22bf3078cc02 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:50.352622 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2b355f-a8ec-45a8-a3b9-22bf3078cc02 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:50.366871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-7f2b355f-a8ec-45a8-a3b9-22bf3078cc02 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:50.366871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:50.366871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:50.366871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:50.366871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:50.366871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:50.366871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:50.366871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:50.366871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:50.366871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:50.366871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:50.366871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:50.366871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:50.366871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:50.366871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:50.367865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:50.367865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:50.367865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:50.367865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:50.367865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:50.367865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:50.367865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:50.367865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:50.367865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:50.367865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:50.367865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:50.367865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:50.367865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:50.367865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:50.367865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:50.367865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:50.367865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:50.368824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:50.368824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:50.368824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:50.368824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:50.368824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:50.368824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:50.368824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:50.368824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:50.371407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 380/756] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:50.372744 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2b355f-a8ec-45a8-a3b9-22bf3078cc02 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:50.411375 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a63466e-6eff-4042-98e0-4e0838ee908b None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:50.478860 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e699ef7b-bf7f-4a83-8e31-f03e5aa7a2d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:50.480008 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e699ef7b-bf7f-4a83-8e31-f03e5aa7a2d2 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:50.480417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e699ef7b-bf7f-4a83-8e31-f03e5aa7a2d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:50.480931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e699ef7b-bf7f-4a83-8e31-f03e5aa7a2d2 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:50.482175 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e699ef7b-bf7f-4a83-8e31-f03e5aa7a2d2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:50.482724 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e699ef7b-bf7f-4a83-8e31-f03e5aa7a2d2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:50.485468 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e699ef7b-bf7f-4a83-8e31-f03e5aa7a2d2 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:50.504455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:50.508695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 377/757] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:15:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:50.510403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e699ef7b-bf7f-4a83-8e31-f03e5aa7a2d2 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:50.516441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2b355f-a8ec-45a8-a3b9-22bf3078cc02 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:50.559752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f54ae2-ed42-42f3-812f-e126bc5706da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:50.560863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f54ae2-ed42-42f3-812f-e126bc5706da None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:50.561914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f54ae2-ed42-42f3-812f-e126bc5706da None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:50.562432 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f54ae2-ed42-42f3-812f-e126bc5706da None admin] PATH_INFO: `/v3/services/network` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:50.563357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f54ae2-ed42-42f3-812f-e126bc5706da None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:50.565049 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f54ae2-ed42-42f3-812f-e126bc5706da None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:50.580011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-b2f54ae2-ed42-42f3-812f-e126bc5706da None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 14:15:50.580011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:50.580011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:15:50.580011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:15:50.580011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:15:50.580011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:15:50.580011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:15:50.580011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:15:50.580011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 14:15:50.580011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:50.580011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:15:50.580011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:50.580011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:50.580011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:50.580011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:50.580011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:50.580011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:50.581316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:50.581316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:50.581316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:50.581316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:50.581316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:50.581316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:50.581316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:15:50.581316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:15:50.581316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:15:50.581316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:15:50.581316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:50.581316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:50.581316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:50.581316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:50.581316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:50.581316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:50.582152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:50.582152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:50.582152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:50.582152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:50.582152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:50.582152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:50.582152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:50.582152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:50.582152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:50.582152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:50.582152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:15:50.582152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:15:50.582152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 14:15:50.582152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:50.585664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 381/758] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:50 2022] GET /identity/v3/services/network => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:50.586800 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f54ae2-ed42-42f3-812f-e126bc5706da None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:50.709745 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e699ef7b-bf7f-4a83-8e31-f03e5aa7a2d2 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:50.718124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f54ae2-ed42-42f3-812f-e126bc5706da None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:50.760882 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:50.761826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f547c7-a834-488c-9bcd-17ae130f623c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:50.763873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f547c7-a834-488c-9bcd-17ae130f623c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:50.764641 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f547c7-a834-488c-9bcd-17ae130f623c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:50.765465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f547c7-a834-488c-9bcd-17ae130f623c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:50.777220 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89843af8-c562-4890-9577-3dd330ea2b0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:50.778304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89843af8-c562-4890-9577-3dd330ea2b0d None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:50.778746 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89843af8-c562-4890-9577-3dd330ea2b0d None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:50.779244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89843af8-c562-4890-9577-3dd330ea2b0d None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:50.780906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f547c7-a834-488c-9bcd-17ae130f623c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:50.781473 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89843af8-c562-4890-9577-3dd330ea2b0d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:50.782013 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89843af8-c562-4890-9577-3dd330ea2b0d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:50.783101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f547c7-a834-488c-9bcd-17ae130f623c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:50.784672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89843af8-c562-4890-9577-3dd330ea2b0d None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:50.802676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-18f547c7-a834-488c-9bcd-17ae130f623c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:50.802676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:50.802676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:50.802676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:50.802676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:50.802676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:50.802676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:50.802676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:50.802676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:50.802676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:50.802676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:50.802676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:50.802676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:50.802676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:50.802676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:50.802676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:50.804023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:50.804023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:50.804023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:50.804023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:50.804023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:50.804023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:50.804023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:50.804023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:50.804023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:50.804023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:50.804023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:50.804023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:50.804023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:50.804023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:50.804023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:50.804023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:50.804023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:50.805032 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:50.805032 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:50.805032 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:50.806834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 382/759] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:50.808051 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f547c7-a834-488c-9bcd-17ae130f623c None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:50.820845 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 378/760] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:50.826684 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89843af8-c562-4890-9577-3dd330ea2b0d None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:50.828444 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:50.944546 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f547c7-a834-488c-9bcd-17ae130f623c None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:50.998527 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626d82a0-3df2-4cdc-ac11-8c7122e0f71b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:50.999460 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626d82a0-3df2-4cdc-ac11-8c7122e0f71b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:50.999916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626d82a0-3df2-4cdc-ac11-8c7122e0f71b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:51.000380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626d82a0-3df2-4cdc-ac11-8c7122e0f71b None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:51.001176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626d82a0-3df2-4cdc-ac11-8c7122e0f71b None admin] RBAC: Adding query filter params (name=network) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:51.001632 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626d82a0-3df2-4cdc-ac11-8c7122e0f71b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:51.003316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626d82a0-3df2-4cdc-ac11-8c7122e0f71b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:51.009279 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 383/761] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Oct 4 14:15:50 2022] GET /identity/v3/services?name=network => generated 124 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:51.010482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626d82a0-3df2-4cdc-ac11-8c7122e0f71b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:51.027133 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89843af8-c562-4890-9577-3dd330ea2b0d None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:51.096292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127054e7-1b41-474a-a98a-d125fa732165 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:51.097221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127054e7-1b41-474a-a98a-d125fa732165 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:51.097730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127054e7-1b41-474a-a98a-d125fa732165 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:51.098038 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127054e7-1b41-474a-a98a-d125fa732165 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:51.099776 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127054e7-1b41-474a-a98a-d125fa732165 None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=5980e353164c4873bcdd2af847e0a1f1) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:51.099776 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127054e7-1b41-474a-a98a-d125fa732165 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:51.101962 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127054e7-1b41-474a-a98a-d125fa732165 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:51.108827 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:51.120723 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 379/762] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:15:50 2022] GET /identity/v3/role_assignments?user.id=5980e353164c4873bcdd2af847e0a1f1&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 259 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:51.122403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127054e7-1b41-474a-a98a-d125fa732165 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:51.140256 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626d82a0-3df2-4cdc-ac11-8c7122e0f71b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:51.185546 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a413fa9e-9d06-4f6e-9aa4-d99f33eebbed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:51.186631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a413fa9e-9d06-4f6e-9aa4-d99f33eebbed None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:51.187187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a413fa9e-9d06-4f6e-9aa4-d99f33eebbed None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:51.187641 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a413fa9e-9d06-4f6e-9aa4-d99f33eebbed None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:51.188536 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a413fa9e-9d06-4f6e-9aa4-d99f33eebbed None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:51.189040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a413fa9e-9d06-4f6e-9aa4-d99f33eebbed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:51.190674 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a413fa9e-9d06-4f6e-9aa4-d99f33eebbed None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:51.209581 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 384/763] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:51.221844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a413fa9e-9d06-4f6e-9aa4-d99f33eebbed None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:51.355199 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a413fa9e-9d06-4f6e-9aa4-d99f33eebbed None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:51.399849 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747e3194-8e53-4bc5-bf37-59bfb4bffb19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:51.400974 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747e3194-8e53-4bc5-bf37-59bfb4bffb19 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:51.401435 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747e3194-8e53-4bc5-bf37-59bfb4bffb19 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:51.401863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747e3194-8e53-4bc5-bf37-59bfb4bffb19 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:51.406278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:51.413688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747e3194-8e53-4bc5-bf37-59bfb4bffb19 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:51.415443 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747e3194-8e53-4bc5-bf37-59bfb4bffb19 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:51.416297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:51.431087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-747e3194-8e53-4bc5-bf37-59bfb4bffb19 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:15:51.431087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:51.431087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:51.431087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:51.431087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:51.431087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:51.431087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:51.431087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:51.431087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:51.431087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:51.431087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:51.431087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:51.431087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:51.431087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:51.431087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:51.431087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:51.432145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:51.433136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:51.433136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:51.433136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:51.433136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:51.433136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:51.433136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:51.433136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:51.433136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:51.433136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:15:51.433136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:51.433688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 385/764] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Oct 4 14:15:51 2022] GET /identity/v3/users/nova => generated 82 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:51.434944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73cc83a-4d9a-486a-8af8-784e5c8eabfe None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:51.435222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73cc83a-4d9a-486a-8af8-784e5c8eabfe None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:51.444342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73cc83a-4d9a-486a-8af8-784e5c8eabfe None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:51.444342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 386/765] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:51.444342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73cc83a-4d9a-486a-8af8-784e5c8eabfe None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:51.525888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127054e7-1b41-474a-a98a-d125fa732165 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:51.615445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654fdd18-7580-4a7c-b774-75978b69046d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:51.616430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73cc83a-4d9a-486a-8af8-784e5c8eabfe None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:51.616796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654fdd18-7580-4a7c-b774-75978b69046d None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:51.617378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654fdd18-7580-4a7c-b774-75978b69046d None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:51.618016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654fdd18-7580-4a7c-b774-75978b69046d None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:51.619926 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654fdd18-7580-4a7c-b774-75978b69046d None admin] RBAC: Adding query filter params (type=network) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:51.622222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654fdd18-7580-4a7c-b774-75978b69046d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:51.625695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654fdd18-7580-4a7c-b774-75978b69046d None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:51.641844 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 380/766] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Oct 4 14:15:51 2022] GET /identity/v3/services?type=network => generated 350 bytes in 520 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:51.645360 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f88a720-d99b-471a-b2a2-c57f739bc115 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:51.646348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f88a720-d99b-471a-b2a2-c57f739bc115 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:51.646908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f88a720-d99b-471a-b2a2-c57f739bc115 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:51.666910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d8a797-974e-4fa5-903c-c80046904797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:51.667836 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d8a797-974e-4fa5-903c-c80046904797 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:51.668339 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d8a797-974e-4fa5-903c-c80046904797 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:51.668844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d8a797-974e-4fa5-903c-c80046904797 None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:51.669784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d8a797-974e-4fa5-903c-c80046904797 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:51.670325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d8a797-974e-4fa5-903c-c80046904797 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:51.672543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d8a797-974e-4fa5-903c-c80046904797 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:51.691249 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-6f88a720-d99b-471a-b2a2-c57f739bc115 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:51.692922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-6f88a720-d99b-471a-b2a2-c57f739bc115 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:51.716011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 387/767] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Oct 4 14:15:51 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:51.717532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d8a797-974e-4fa5-903c-c80046904797 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:51.807031 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 381/768] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:51 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:15:51.808226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f88a720-d99b-471a-b2a2-c57f739bc115 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:51.841569 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:51.883000 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:51.938393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d8a797-974e-4fa5-903c-c80046904797 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:51.985965 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae7f2a2-b175-40be-9987-68f7265654c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:51.986589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae7f2a2-b175-40be-9987-68f7265654c8 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:51.986902 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae7f2a2-b175-40be-9987-68f7265654c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:51.987180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae7f2a2-b175-40be-9987-68f7265654c8 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:51.987904 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae7f2a2-b175-40be-9987-68f7265654c8 None admin] RBAC: Adding query filter params (interface=public, service_id=813c60c380624f5f903a98b9338bba3a, region_id=RegionOne) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:51.988210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae7f2a2-b175-40be-9987-68f7265654c8 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:51.989765 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae7f2a2-b175-40be-9987-68f7265654c8 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:51.998432 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f88a720-d99b-471a-b2a2-c57f739bc115 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:52.006611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 388/769] 10.222.0.64 () {56 vars in 1251 bytes} [Tue Oct 4 14:15:51 2022] GET /identity/v3/endpoints?service_id=813c60c380624f5f903a98b9338bba3a&interface=public®ion_id=RegionOne => generated 194 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:52.008100 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9066ff8-52a7-4a99-85e4-0a3494770c0b None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:52.008416 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9066ff8-52a7-4a99-85e4-0a3494770c0b None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:52.008669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9066ff8-52a7-4a99-85e4-0a3494770c0b None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:52.010319 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 389/770] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:52.011101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9066ff8-52a7-4a99-85e4-0a3494770c0b None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:52.071234 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7a8aa0-381d-496f-926f-e0fa8deb3ff1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:52.071896 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7a8aa0-381d-496f-926f-e0fa8deb3ff1 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:52.072173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7a8aa0-381d-496f-926f-e0fa8deb3ff1 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:52.072661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7a8aa0-381d-496f-926f-e0fa8deb3ff1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:52.087601 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7a8aa0-381d-496f-926f-e0fa8deb3ff1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:52.089408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7a8aa0-381d-496f-926f-e0fa8deb3ff1 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:52.109063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-dc7a8aa0-381d-496f-926f-e0fa8deb3ff1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:52.109063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:52.109063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:52.109063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:52.109063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:52.109063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:52.109063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:52.109063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:52.109063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:52.109063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:52.109063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:52.109063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:52.109063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:52.109063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:52.109063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:52.109063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:52.112793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:52.112793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:52.112793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:52.112793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:52.112793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:52.112793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:52.112793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:52.112793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:52.112793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:52.112793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:52.112793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:52.112793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:52.112793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:52.112793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:52.112793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:52.112793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:52.112793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:52.116367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:52.116367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:52.116367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:52.116367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 382/771] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:52.116367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d03c3ec-2a29-4fff-b2e1-f7e5eed1e917 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:52.116367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d03c3ec-2a29-4fff-b2e1-f7e5eed1e917 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:52.116367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d03c3ec-2a29-4fff-b2e1-f7e5eed1e917 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:52.155882 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-8d03c3ec-2a29-4fff-b2e1-f7e5eed1e917 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:52.158773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8d03c3ec-2a29-4fff-b2e1-f7e5eed1e917 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:52.185509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9066ff8-52a7-4a99-85e4-0a3494770c0b None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:52.210855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:52.237582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5535cc37-e379-4ff6-80fe-c50ad335709e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:52.238670 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5535cc37-e379-4ff6-80fe-c50ad335709e None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:52.239153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5535cc37-e379-4ff6-80fe-c50ad335709e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:52.239613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5535cc37-e379-4ff6-80fe-c50ad335709e None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:52.240509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5535cc37-e379-4ff6-80fe-c50ad335709e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:52.242126 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5535cc37-e379-4ff6-80fe-c50ad335709e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:52.256903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 390/772] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Oct 4 14:15:52 2022] GET /identity/v3/services => generated 772 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:52.258208 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5535cc37-e379-4ff6-80fe-c50ad335709e None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:52.331020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 383/773] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:52 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:15:52.440959 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5535cc37-e379-4ff6-80fe-c50ad335709e None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:52.443051 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d03c3ec-2a29-4fff-b2e1-f7e5eed1e917 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:52.501745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3d1f71-d90b-45cd-a290-c123b82808e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:52.502370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3d1f71-d90b-45cd-a290-c123b82808e1 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:52.502642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3d1f71-d90b-45cd-a290-c123b82808e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:52.502911 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3d1f71-d90b-45cd-a290-c123b82808e1 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:52.503574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3d1f71-d90b-45cd-a290-c123b82808e1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:52.503872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3d1f71-d90b-45cd-a290-c123b82808e1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:52.505337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3d1f71-d90b-45cd-a290-c123b82808e1 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:52.533797 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:52.538679 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 391/774] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:52.549355 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3d1f71-d90b-45cd-a290-c123b82808e1 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:52.647130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:52.722497 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d03c3ec-2a29-4fff-b2e1-f7e5eed1e917 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:52.745476 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3d1f71-d90b-45cd-a290-c123b82808e1 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:52.802142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775b70ef-be26-495f-addf-1c274b98abeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:52.802802 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775b70ef-be26-495f-addf-1c274b98abeb None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:52.803080 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775b70ef-be26-495f-addf-1c274b98abeb None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:52.803352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775b70ef-be26-495f-addf-1c274b98abeb None admin] PATH_INFO: `/v3/roles/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:52.819582 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775b70ef-be26-495f-addf-1c274b98abeb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:52.821374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0af849-4927-4395-95dc-a739e8bbd97c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:52.822019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0af849-4927-4395-95dc-a739e8bbd97c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:52.822288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0af849-4927-4395-95dc-a739e8bbd97c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:52.822729 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0af849-4927-4395-95dc-a739e8bbd97c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:52.825608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-775b70ef-be26-495f-addf-1c274b98abeb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:52.825608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:52.825608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:52.825608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:52.825608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:52.825608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:52.825608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:52.825608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:52.825608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:52.825608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:52.825608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:52.825608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:52.825608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:52.825608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:52.825608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:52.825608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:52.828127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:52.828127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:52.828127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:52.828127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:52.828127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:52.828127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:52.828127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:52.828127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:52.828127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:52.828127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:52.828127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:52.828127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:52.828127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:52.828127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:52.828127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:52.828127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:52.829665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:52.829665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:52.829665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:52.829665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:52.829665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:52.829665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:52.829665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:52.829665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:52.829665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:52.829665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:52.829665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:15:52.829665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:52.832812 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 384/775] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 391 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:52.833954 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775b70ef-be26-495f-addf-1c274b98abeb None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:52.840262 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0af849-4927-4395-95dc-a739e8bbd97c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:52.842254 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0af849-4927-4395-95dc-a739e8bbd97c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:52.863884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-1c0af849-4927-4395-95dc-a739e8bbd97c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:52.863884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:52.863884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:52.863884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:52.863884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:52.863884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:52.863884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:52.863884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:52.863884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:52.863884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:52.863884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:52.863884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:52.863884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:52.863884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:52.863884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:52.865096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:52.865096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:52.865096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:52.865096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:52.865096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:52.865096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:52.865096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:52.865096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:52.865096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:52.865096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:52.865096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:52.865096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:52.865096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:52.865096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:52.865096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:52.865096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:52.865096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:52.866191 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:52.866191 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:52.866191 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:52.866191 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:52.866191 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:52.866191 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:52.866191 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:52.866191 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:52.867411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 392/776] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:52.868374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0af849-4927-4395-95dc-a739e8bbd97c None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:52.972117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:52.994617 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775b70ef-be26-495f-addf-1c274b98abeb None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:53.012539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0af849-4927-4395-95dc-a739e8bbd97c None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:53.015127 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:53.056948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b3fe8e-ebb5-43a0-a9c6-3ba92b071bbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:53.057692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b3fe8e-ebb5-43a0-a9c6-3ba92b071bbc None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:53.058037 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b3fe8e-ebb5-43a0-a9c6-3ba92b071bbc None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:53.058378 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b3fe8e-ebb5-43a0-a9c6-3ba92b071bbc None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:53.059128 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b3fe8e-ebb5-43a0-a9c6-3ba92b071bbc None admin] RBAC: Adding query filter params (name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:53.059955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b3fe8e-ebb5-43a0-a9c6-3ba92b071bbc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:53.062357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b3fe8e-ebb5-43a0-a9c6-3ba92b071bbc None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:53.068745 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c667468-ea97-4e44-82ae-19543865a531 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:53.070165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c667468-ea97-4e44-82ae-19543865a531 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:53.070875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c667468-ea97-4e44-82ae-19543865a531 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:53.071533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c667468-ea97-4e44-82ae-19543865a531 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:53.072782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c667468-ea97-4e44-82ae-19543865a531 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:53.075467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c667468-ea97-4e44-82ae-19543865a531 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:53.077273 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 393/777] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:15:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:53.078495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b3fe8e-ebb5-43a0-a9c6-3ba92b071bbc None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:53.094774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-1c667468-ea97-4e44-82ae-19543865a531 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 14:15:53.094774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:53.094774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:15:53.094774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:15:53.094774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:15:53.094774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:15:53.094774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:15:53.094774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:15:53.094774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 14:15:53.094774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:53.094774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:15:53.094774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:53.094774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:53.094774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:53.094774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:53.094774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:53.094774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:53.097117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:53.097117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:53.097117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:53.097117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:53.097117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:53.097117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:53.097117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:15:53.097117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:15:53.097117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:15:53.097117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:15:53.097117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:53.097117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:53.097117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:53.097117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:53.097117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:53.097117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:53.099067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:53.099067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:53.099067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:53.099067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:53.099067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:53.099067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:53.099067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:53.099067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:53.099067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:53.099067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:53.099067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:15:53.099067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:15:53.099067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 14:15:53.099067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:53.103477 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 385/778] 10.222.0.64 () {56 vars in 1129 bytes} [Tue Oct 4 14:15:52 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:53.105394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c667468-ea97-4e44-82ae-19543865a531 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:53.252714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c667468-ea97-4e44-82ae-19543865a531 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:53.270109 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b3fe8e-ebb5-43a0-a9c6-3ba92b071bbc None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:53.300767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfe466b-21cb-4d3b-bc91-a61ef4ca8bae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:53.301436 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfe466b-21cb-4d3b-bc91-a61ef4ca8bae None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:53.301718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfe466b-21cb-4d3b-bc91-a61ef4ca8bae None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:53.301996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfe466b-21cb-4d3b-bc91-a61ef4ca8bae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:53.317307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfe466b-21cb-4d3b-bc91-a61ef4ca8bae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:53.319610 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfe466b-21cb-4d3b-bc91-a61ef4ca8bae None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:53.332967 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da5c3bf-1c54-46f8-a268-de9c9224cb26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:53.333985 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da5c3bf-1c54-46f8-a268-de9c9224cb26 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:53.334449 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da5c3bf-1c54-46f8-a268-de9c9224cb26 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:53.334915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da5c3bf-1c54-46f8-a268-de9c9224cb26 None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:53.335813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da5c3bf-1c54-46f8-a268-de9c9224cb26 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:53.336351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da5c3bf-1c54-46f8-a268-de9c9224cb26 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:53.338127 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da5c3bf-1c54-46f8-a268-de9c9224cb26 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:53.338887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-4bfe466b-21cb-4d3b-bc91-a61ef4ca8bae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:53.338887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:53.338887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:53.338887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:53.338887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:53.338887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:53.338887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:53.338887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:53.338887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:53.338887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:53.338887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:53.338887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:53.338887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:53.338887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:53.338887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:53.338887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:53.339823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:53.339823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:53.339823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:53.339823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:53.339823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:53.339823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:53.339823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:53.339823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:53.339823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:53.339823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:53.339823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:53.339823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:53.339823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:53.339823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:53.339823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:53.339823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:53.339823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:53.340784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:53.340784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:53.340784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:53.344416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 386/779] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:53.346809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfe466b-21cb-4d3b-bc91-a61ef4ca8bae None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:53.351218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:53.367721 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 394/780] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:53.369265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da5c3bf-1c54-46f8-a268-de9c9224cb26 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:53.409148 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:53.478650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfe466b-21cb-4d3b-bc91-a61ef4ca8bae None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:53.519940 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da5c3bf-1c54-46f8-a268-de9c9224cb26 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:53.524991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c82b8b-c3c8-4ede-bf1c-08aad315e434 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:53.525758 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c82b8b-c3c8-4ede-bf1c-08aad315e434 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:53.526106 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c82b8b-c3c8-4ede-bf1c-08aad315e434 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:53.526448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c82b8b-c3c8-4ede-bf1c-08aad315e434 None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:53.527168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c82b8b-c3c8-4ede-bf1c-08aad315e434 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:53.527605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c82b8b-c3c8-4ede-bf1c-08aad315e434 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:53.529140 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c82b8b-c3c8-4ede-bf1c-08aad315e434 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:53.540218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 387/781] 10.222.0.64 () {56 vars in 1124 bytes} [Tue Oct 4 14:15:53 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:53.541090 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c82b8b-c3c8-4ede-bf1c-08aad315e434 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:53.565472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a595789-2ab5-464d-b01d-ca0a042afaeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:53.566412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a595789-2ab5-464d-b01d-ca0a042afaeb None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:53.566874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a595789-2ab5-464d-b01d-ca0a042afaeb None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:53.567353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a595789-2ab5-464d-b01d-ca0a042afaeb None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:53.568385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a595789-2ab5-464d-b01d-ca0a042afaeb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:53.568924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a595789-2ab5-464d-b01d-ca0a042afaeb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:53.570563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a595789-2ab5-464d-b01d-ca0a042afaeb None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:53.585448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 395/782] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:53.586717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a595789-2ab5-464d-b01d-ca0a042afaeb None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:53.634427 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:53.677648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c82b8b-c3c8-4ede-bf1c-08aad315e434 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:53.719753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd54a261-1199-47f0-aea6-8b6fa0b98a9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:53.720591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd54a261-1199-47f0-aea6-8b6fa0b98a9f None admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:53.720944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd54a261-1199-47f0-aea6-8b6fa0b98a9f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:53.721312 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd54a261-1199-47f0-aea6-8b6fa0b98a9f None admin] PATH_INFO: `/v3/projects/895ece8b70ef44adadc3416f81a80731/users/03081d7d581745c88c12e9d089fcb2d3/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:53.723838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd54a261-1199-47f0-aea6-8b6fa0b98a9f None admin] RBAC: Authorizing `identity:create_grant(project_id=895ece8b70ef44adadc3416f81a80731, user_id=03081d7d581745c88c12e9d089fcb2d3, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:53.725624 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd54a261-1199-47f0-aea6-8b6fa0b98a9f None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:53.743033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 388/783] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:15:53 2022] PUT /identity/v3/projects/895ece8b70ef44adadc3416f81a80731/users/03081d7d581745c88c12e9d089fcb2d3/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:15:53.744294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd54a261-1199-47f0-aea6-8b6fa0b98a9f None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:53.836480 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:53.860161 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a595789-2ab5-464d-b01d-ca0a042afaeb None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:53.879859 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd54a261-1199-47f0-aea6-8b6fa0b98a9f None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:53.907201 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8db7c7-d0b4-4a02-a5c6-d5bd971e4c2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:53.907838 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8db7c7-d0b4-4a02-a5c6-d5bd971e4c2e None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:53.908119 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8db7c7-d0b4-4a02-a5c6-d5bd971e4c2e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:53.908427 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8db7c7-d0b4-4a02-a5c6-d5bd971e4c2e None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:53.909291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8db7c7-d0b4-4a02-a5c6-d5bd971e4c2e None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:53.909601 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8db7c7-d0b4-4a02-a5c6-d5bd971e4c2e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:53.911058 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8db7c7-d0b4-4a02-a5c6-d5bd971e4c2e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:53.917293 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:53.923140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 396/784] 10.222.0.64 () {56 vars in 1124 bytes} [Tue Oct 4 14:15:53 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 332 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:53.923140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dde16e-18e7-4a2a-b82c-cab3ec1f5492 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:53.923140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dde16e-18e7-4a2a-b82c-cab3ec1f5492 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:53.923140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dde16e-18e7-4a2a-b82c-cab3ec1f5492 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:53.923140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 397/785] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:53.928267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dde16e-18e7-4a2a-b82c-cab3ec1f5492 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:53.937563 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d5b782-c7b1-4639-b021-6d1b6bf23ee1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:53.938653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d5b782-c7b1-4639-b021-6d1b6bf23ee1 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:53.939173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d5b782-c7b1-4639-b021-6d1b6bf23ee1 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:53.939653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d5b782-c7b1-4639-b021-6d1b6bf23ee1 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:53.952732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d5b782-c7b1-4639-b021-6d1b6bf23ee1 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:53.955731 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d5b782-c7b1-4639-b021-6d1b6bf23ee1 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:53.974221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-d5d5b782-c7b1-4639-b021-6d1b6bf23ee1 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 14:15:53.974221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:53.974221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:53.974221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:53.974221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:53.974221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:53.974221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:53.974221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:53.974221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:53.974221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:53.974221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:53.974221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:53.974221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:53.974221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:53.974221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:53.974221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:53.975268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:53.976252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:53.976252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:53.976252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:53.976252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:53.976252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:53.976252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:53.976252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:53.976252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:53.976252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 14:15:53.976252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:53.983723 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 389/786] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Oct 4 14:15:53 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:53.983723 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e28acb1-8449-4c64-9e79-28f131af0947 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:53.983723 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e28acb1-8449-4c64-9e79-28f131af0947 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:53.983723 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e28acb1-8449-4c64-9e79-28f131af0947 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:54.006067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-3e28acb1-8449-4c64-9e79-28f131af0947 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:54.007348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-3e28acb1-8449-4c64-9e79-28f131af0947 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:54.083801 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dde16e-18e7-4a2a-b82c-cab3ec1f5492 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:54.102884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 390/787] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:53 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:15:54.103682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e28acb1-8449-4c64-9e79-28f131af0947 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:54.132170 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b83559c-577f-4b16-b891-6bd829780101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:54.133228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b83559c-577f-4b16-b891-6bd829780101 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:54.133711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b83559c-577f-4b16-b891-6bd829780101 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:54.134180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b83559c-577f-4b16-b891-6bd829780101 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:54.135390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b83559c-577f-4b16-b891-6bd829780101 None admin] RBAC: Adding query filter params (interface=public, service_id=45b206b3897e4bbab6bb6e547cda2d8c, region_id=RegionOne) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:54.135805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b83559c-577f-4b16-b891-6bd829780101 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:54.137330 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b83559c-577f-4b16-b891-6bd829780101 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:54.148792 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 398/788] 10.222.0.64 () {56 vars in 1251 bytes} [Tue Oct 4 14:15:53 2022] GET /identity/v3/endpoints?service_id=45b206b3897e4bbab6bb6e547cda2d8c&interface=public®ion_id=RegionOne => generated 194 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:54.152444 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b83559c-577f-4b16-b891-6bd829780101 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:54.171626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:54.249657 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e28acb1-8449-4c64-9e79-28f131af0947 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:54.270425 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:54.326488 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08574271-d7ac-40ef-8b6a-f470e87632d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:54.327622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08574271-d7ac-40ef-8b6a-f470e87632d5 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:54.328167 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08574271-d7ac-40ef-8b6a-f470e87632d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:54.328701 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08574271-d7ac-40ef-8b6a-f470e87632d5 None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:54.330250 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08574271-d7ac-40ef-8b6a-f470e87632d5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:54.333092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08574271-d7ac-40ef-8b6a-f470e87632d5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:54.335682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08574271-d7ac-40ef-8b6a-f470e87632d5 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:54.364428 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b83559c-577f-4b16-b891-6bd829780101 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:54.370949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 391/789] 10.222.0.64 () {56 vars in 1149 bytes} [Tue Oct 4 14:15:54 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:54.377847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08574271-d7ac-40ef-8b6a-f470e87632d5 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:54.419514 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304f1dcb-24c4-4a5f-b6f4-352a5f488289 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:54.420678 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304f1dcb-24c4-4a5f-b6f4-352a5f488289 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:54.421592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304f1dcb-24c4-4a5f-b6f4-352a5f488289 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:54.421956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304f1dcb-24c4-4a5f-b6f4-352a5f488289 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:54.422912 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304f1dcb-24c4-4a5f-b6f4-352a5f488289 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:54.424862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304f1dcb-24c4-4a5f-b6f4-352a5f488289 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:54.440354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 399/790] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Oct 4 14:15:54 2022] GET /identity/v3/services => generated 772 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:15:54.532040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304f1dcb-24c4-4a5f-b6f4-352a5f488289 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:54.621982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08574271-d7ac-40ef-8b6a-f470e87632d5 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:54.707760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c64d5c9-beba-4473-8f59-29a290fec4de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:54.710238 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c64d5c9-beba-4473-8f59-29a290fec4de None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:54.710780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c64d5c9-beba-4473-8f59-29a290fec4de None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:54.711322 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c64d5c9-beba-4473-8f59-29a290fec4de None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:54.727911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c64d5c9-beba-4473-8f59-29a290fec4de None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:54.729700 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c64d5c9-beba-4473-8f59-29a290fec4de None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:54.747014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:54.755639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-6c64d5c9-beba-4473-8f59-29a290fec4de None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:54.755639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:54.755639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:54.755639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:54.755639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:54.755639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:54.755639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:54.755639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:54.755639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:54.755639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:54.755639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:54.755639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:54.755639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:54.755639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:54.755639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:54.755639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:54.757885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:54.757885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:54.757885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:54.757885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:54.757885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:54.757885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:54.757885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:54.757885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:54.757885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:54.757885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:54.757885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:54.757885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:54.757885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:54.757885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:54.757885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:54.757885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:54.757885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:54.760026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:54.760026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:54.760026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:54.760026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 392/791] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 380 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:54.764585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c64d5c9-beba-4473-8f59-29a290fec4de None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:54.800166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304f1dcb-24c4-4a5f-b6f4-352a5f488289 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:54.829729 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:54.851597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbbe5f9-0af5-408d-bd7f-8d4827f63d6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:54.852463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbbe5f9-0af5-408d-bd7f-8d4827f63d6d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:54.852842 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbbe5f9-0af5-408d-bd7f-8d4827f63d6d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:54.853215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbbe5f9-0af5-408d-bd7f-8d4827f63d6d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:54.866255 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbbe5f9-0af5-408d-bd7f-8d4827f63d6d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:54.871542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-7dbbe5f9-0af5-408d-bd7f-8d4827f63d6d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:15:54.871542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:54.871542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:54.871542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:54.871542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:54.871542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:54.871542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:54.871542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:54.871542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:54.871542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:54.871542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:54.871542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:54.871542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:54.871542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:54.871542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:54.871542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:54.873594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:54.873594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:54.873594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:54.873594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:54.873594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:54.873594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:54.873594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:54.873594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:54.873594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:54.873594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:54.873594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:54.873594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:54.873594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:54.873594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:54.873594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:54.873594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:54.875298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:54.875298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:54.875298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:54.875298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:54.875298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:54.875298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:54.875298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:54.875298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:54.875298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:54.875298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:54.875298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:15:54.875298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:54.875298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 400/792] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:15:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 343 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:54.883889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbbe5f9-0af5-408d-bd7f-8d4827f63d6d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:54.975381 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c64d5c9-beba-4473-8f59-29a290fec4de None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:55.028948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbbe5f9-0af5-408d-bd7f-8d4827f63d6d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:55.060197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09fea28-6707-4403-a318-ab8f59be9fe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:55.060949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09fea28-6707-4403-a318-ab8f59be9fe3 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:55.061229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09fea28-6707-4403-a318-ab8f59be9fe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:55.061501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09fea28-6707-4403-a318-ab8f59be9fe3 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:55.062392 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09fea28-6707-4403-a318-ab8f59be9fe3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:55.062798 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09fea28-6707-4403-a318-ab8f59be9fe3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:55.064235 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09fea28-6707-4403-a318-ab8f59be9fe3 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:55.081161 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c72e8a-d0df-4b23-85e3-38fbbbe5136f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:55.081798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c72e8a-d0df-4b23-85e3-38fbbbe5136f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:55.082176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c72e8a-d0df-4b23-85e3-38fbbbe5136f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:55.082591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c72e8a-d0df-4b23-85e3-38fbbbe5136f None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:55.083390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c72e8a-d0df-4b23-85e3-38fbbbe5136f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:55.083702 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c72e8a-d0df-4b23-85e3-38fbbbe5136f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:55.085440 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c72e8a-d0df-4b23-85e3-38fbbbe5136f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:55.098302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 393/793] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 336 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:15:55.101470 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09fea28-6707-4403-a318-ab8f59be9fe3 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:55.105691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 401/794] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:15:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:15:55.107946 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:55.115322 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c72e8a-d0df-4b23-85e3-38fbbbe5136f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:55.323797 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c72e8a-d0df-4b23-85e3-38fbbbe5136f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:55.324655 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:55.337578 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09fea28-6707-4403-a318-ab8f59be9fe3 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:55.370950 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb1554c-698e-4cde-bc73-06a886effea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:55.371606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb1554c-698e-4cde-bc73-06a886effea9 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:55.371880 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb1554c-698e-4cde-bc73-06a886effea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:55.373252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb1554c-698e-4cde-bc73-06a886effea9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:55.399315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb1554c-698e-4cde-bc73-06a886effea9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:55.401776 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb1554c-698e-4cde-bc73-06a886effea9 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:55.430578 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992db05b-b650-4e41-ba79-56a082478296 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:55.431235 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992db05b-b650-4e41-ba79-56a082478296 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:55.431519 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992db05b-b650-4e41-ba79-56a082478296 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:55.431796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992db05b-b650-4e41-ba79-56a082478296 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:55.434235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-1fb1554c-698e-4cde-bc73-06a886effea9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:55.434235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:55.434235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:55.434235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:55.434235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:55.434235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:55.434235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:55.434235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:55.434235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:55.434235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:55.434235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:55.434235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:55.434235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:55.434235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:55.434235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:55.434235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:55.436379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:55.436379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:55.436379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:55.436379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:55.436379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:55.436379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:55.436379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:55.436379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:55.436379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:55.436379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:55.436379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:55.436379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:55.436379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:55.436379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:55.436379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:55.436379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:55.436379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:55.438599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:55.438599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:55.438599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:55.444107 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 402/795] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:55.444107 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb1554c-698e-4cde-bc73-06a886effea9 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:55.450542 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992db05b-b650-4e41-ba79-56a082478296 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:55.453059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992db05b-b650-4e41-ba79-56a082478296 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:55.477485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-992db05b-b650-4e41-ba79-56a082478296 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:55.477485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:55.477485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:55.477485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:55.477485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:55.477485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:55.477485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:55.477485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:55.477485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:55.477485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:55.477485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:55.477485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:55.477485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:55.477485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:55.477485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:55.479535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:55.479535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:55.479535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:55.479535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:55.479535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:55.479535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:55.479535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:55.479535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:55.479535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:55.479535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:55.479535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:55.479535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:55.479535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:55.479535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:55.479535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:55.479535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:55.479535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:55.481643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:55.481643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:55.481643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:55.481643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:55.481643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:55.481643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:55.481643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:55.481643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:55.488360 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 394/796] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 384 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:55.492296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992db05b-b650-4e41-ba79-56a082478296 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:55.570544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:55.760897 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb1554c-698e-4cde-bc73-06a886effea9 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:55.847715 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992db05b-b650-4e41-ba79-56a082478296 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:55.903452 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f510cd-c059-4fa8-942a-c44f2a6f8205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:55.905339 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f510cd-c059-4fa8-942a-c44f2a6f8205 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:55.905822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f510cd-c059-4fa8-942a-c44f2a6f8205 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:55.906307 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f510cd-c059-4fa8-942a-c44f2a6f8205 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:55.907583 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f510cd-c059-4fa8-942a-c44f2a6f8205 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:55.908091 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f510cd-c059-4fa8-942a-c44f2a6f8205 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:55.917787 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f510cd-c059-4fa8-942a-c44f2a6f8205 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:55.949017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 403/797] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 507 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:55.955030 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f510cd-c059-4fa8-942a-c44f2a6f8205 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:55.967825 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24573f8-31a8-42cf-afd3-bd603286cd42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:55.968864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24573f8-31a8-42cf-afd3-bd603286cd42 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:55.969342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24573f8-31a8-42cf-afd3-bd603286cd42 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:55.970272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24573f8-31a8-42cf-afd3-bd603286cd42 None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:55.971237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24573f8-31a8-42cf-afd3-bd603286cd42 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:55.973787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24573f8-31a8-42cf-afd3-bd603286cd42 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:55.975487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24573f8-31a8-42cf-afd3-bd603286cd42 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:56.002892 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 395/798] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 511 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:56.008869 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24573f8-31a8-42cf-afd3-bd603286cd42 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:56.035626 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:56.142373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f510cd-c059-4fa8-942a-c44f2a6f8205 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:56.158169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24573f8-31a8-42cf-afd3-bd603286cd42 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:56.168559 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:56.205618 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93e5c78-2004-4f1f-b414-dca040dd428a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:56.206653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e5c78-2004-4f1f-b414-dca040dd428a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:56.207128 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e5c78-2004-4f1f-b414-dca040dd428a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:56.207796 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e5c78-2004-4f1f-b414-dca040dd428a None admin] PATH_INFO: `/v3/users/swift` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:56.218666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93e5c78-2004-4f1f-b414-dca040dd428a None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:56.220701 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93e5c78-2004-4f1f-b414-dca040dd428a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:56.223325 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5252a2fe-f2a1-4bd2-9598-594b81cac192 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:56.224671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5252a2fe-f2a1-4bd2-9598-594b81cac192 None admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:56.225290 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5252a2fe-f2a1-4bd2-9598-594b81cac192 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:56.225725 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5252a2fe-f2a1-4bd2-9598-594b81cac192 None admin] PATH_INFO: `/v3/projects/895ece8b70ef44adadc3416f81a80731/users/34afa4aac10c4a018ca05a0ba34c940a/roles/3ebe3c0be8c34c5eae533311e9b44176` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:56.236639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-b93e5c78-2004-4f1f-b414-dca040dd428a None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 14:15:56.236639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:56.236639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:56.236639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:56.236639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:56.236639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:56.236639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:56.236639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:56.236639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:56.236639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:56.236639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:56.236639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:56.236639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:56.236639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:56.236639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:56.236639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:56.237761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:56.238860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:56.238860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:56.238860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:56.238860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:56.238860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:56.238860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:56.238860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:56.238860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:56.238860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 14:15:56.238860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:56.241091 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 404/799] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:15:55 2022] GET /identity/v3/users/swift => generated 83 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:56.242533 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d74c696-a9d1-4719-a8f6-f34995c3ed82 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:56.242984 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d74c696-a9d1-4719-a8f6-f34995c3ed82 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:56.243396 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d74c696-a9d1-4719-a8f6-f34995c3ed82 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:56.245404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 405/800] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:56.247072 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d74c696-a9d1-4719-a8f6-f34995c3ed82 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:56.257944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5252a2fe-f2a1-4bd2-9598-594b81cac192 None admin] RBAC: Authorizing `identity:create_grant(project_id=895ece8b70ef44adadc3416f81a80731, user_id=34afa4aac10c4a018ca05a0ba34c940a, role_id=3ebe3c0be8c34c5eae533311e9b44176)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:56.260944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5252a2fe-f2a1-4bd2-9598-594b81cac192 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:56.291954 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 396/801] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:15:56 2022] PUT /identity/v3/projects/895ece8b70ef44adadc3416f81a80731/users/34afa4aac10c4a018ca05a0ba34c940a/roles/3ebe3c0be8c34c5eae533311e9b44176 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:15:56.294940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac0dc86-57f5-408d-b930-9d6a65c7b077 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:56.296586 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac0dc86-57f5-408d-b930-9d6a65c7b077 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:56.297704 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac0dc86-57f5-408d-b930-9d6a65c7b077 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:56.356714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-8ac0dc86-57f5-408d-b930-9d6a65c7b077 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:56.359412 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8ac0dc86-57f5-408d-b930-9d6a65c7b077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:56.517780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:56.547360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d74c696-a9d1-4719-a8f6-f34995c3ed82 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:56.556973 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 397/802] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:56 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:15:56.603759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02fad664-639e-4dd9-9687-a0024cd429cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:56.604441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fad664-639e-4dd9-9687-a0024cd429cd None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:56.604715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fad664-639e-4dd9-9687-a0024cd429cd None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:56.604979 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fad664-639e-4dd9-9687-a0024cd429cd None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:56.605726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02fad664-639e-4dd9-9687-a0024cd429cd None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:56.606030 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02fad664-639e-4dd9-9687-a0024cd429cd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:56.608011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02fad664-639e-4dd9-9687-a0024cd429cd None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:56.626301 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdc2ec9-f953-484c-9ebe-4a6485beee25 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:56.626820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdc2ec9-f953-484c-9ebe-4a6485beee25 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:56.627325 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdc2ec9-f953-484c-9ebe-4a6485beee25 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:56.630452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 398/803] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:56 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:56.632645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 406/804] 10.222.0.64 () {56 vars in 1135 bytes} [Tue Oct 4 14:15:56 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 386 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:56.639645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdc2ec9-f953-484c-9ebe-4a6485beee25 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:56.643471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae04decf-209d-497d-b2da-9ed2949330ab None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:56.644069 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae04decf-209d-497d-b2da-9ed2949330ab None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:56.644623 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae04decf-209d-497d-b2da-9ed2949330ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:56.666987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-ae04decf-209d-497d-b2da-9ed2949330ab None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:56.668703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ae04decf-209d-497d-b2da-9ed2949330ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:56.702456 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:56.791931 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 407/805] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:56 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:15:56.875226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdc2ec9-f953-484c-9ebe-4a6485beee25 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:56.940189 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53528d37-7e3a-4876-9b07-e605d012bad0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:56.941193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53528d37-7e3a-4876-9b07-e605d012bad0 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:56.941601 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53528d37-7e3a-4876-9b07-e605d012bad0 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:56.942024 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53528d37-7e3a-4876-9b07-e605d012bad0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:56.960507 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53528d37-7e3a-4876-9b07-e605d012bad0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:56.963401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53528d37-7e3a-4876-9b07-e605d012bad0 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:56.989007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-53528d37-7e3a-4876-9b07-e605d012bad0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:56.989007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:56.989007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:56.989007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:56.989007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:56.989007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:56.989007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:56.989007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:56.989007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:56.989007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:56.989007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:56.989007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:56.989007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:56.989007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:56.989007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:56.989007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:56.990023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:56.990023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:56.990023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:56.990023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:56.990023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:56.990023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:56.990023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:56.990023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:56.990023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:56.990023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:56.990023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:56.990023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:56.990023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:56.990023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:56.990023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:56.990023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:56.990023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:56.990927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:56.990927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:56.990927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:56.990927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 399/806] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 352 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:57.001556 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae04decf-209d-497d-b2da-9ed2949330ab None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:57.022379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53528d37-7e3a-4876-9b07-e605d012bad0 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:57.141670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:57.245315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae04decf-209d-497d-b2da-9ed2949330ab None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:57.293637 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f92f1c-c454-40c0-9057-cce758c6f900 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:57.294644 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f92f1c-c454-40c0-9057-cce758c6f900 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:57.295173 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f92f1c-c454-40c0-9057-cce758c6f900 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:57.296404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f92f1c-c454-40c0-9057-cce758c6f900 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:57.297417 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f92f1c-c454-40c0-9057-cce758c6f900 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:57.297944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f92f1c-c454-40c0-9057-cce758c6f900 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:57.299568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f92f1c-c454-40c0-9057-cce758c6f900 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:57.303553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53528d37-7e3a-4876-9b07-e605d012bad0 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:57.320254 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 408/807] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:57.321766 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f92f1c-c454-40c0-9057-cce758c6f900 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:57.347426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:57.395949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed00e17-05ff-4e43-8d49-ec2bca91bf2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:57.397246 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed00e17-05ff-4e43-8d49-ec2bca91bf2f None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:57.397750 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed00e17-05ff-4e43-8d49-ec2bca91bf2f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:57.398224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed00e17-05ff-4e43-8d49-ec2bca91bf2f None admin] PATH_INFO: `/v3/services/network` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:57.399809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed00e17-05ff-4e43-8d49-ec2bca91bf2f None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:57.403358 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed00e17-05ff-4e43-8d49-ec2bca91bf2f None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:57.420877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-3ed00e17-05ff-4e43-8d49-ec2bca91bf2f None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 14:15:57.420877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:57.420877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:15:57.420877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:15:57.420877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:15:57.420877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:15:57.420877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:15:57.420877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:15:57.420877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 14:15:57.420877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:57.420877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:15:57.420877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:57.420877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:57.420877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:57.420877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:57.420877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:57.420877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:57.422196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:57.422196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:57.422196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:57.422196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:57.422196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:57.422196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:57.422196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:15:57.422196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:15:57.422196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:15:57.422196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:15:57.422196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:57.422196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:57.422196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:57.422196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:57.422196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:57.422196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:57.423059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:57.423059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:57.423059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:57.423059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:57.423059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:57.423059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:57.423059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:57.423059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:57.423059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:57.423059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:57.423059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:15:57.423059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:15:57.423059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 14:15:57.423059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:57.426386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 400/808] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:57 2022] GET /identity/v3/services/network => generated 88 bytes in 405 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:57.427653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed00e17-05ff-4e43-8d49-ec2bca91bf2f None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:57.521535 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f92f1c-c454-40c0-9057-cce758c6f900 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:57.587032 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32346370-9e22-4105-afb9-031ab0f93406 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:57.588187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32346370-9e22-4105-afb9-031ab0f93406 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:57.588748 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32346370-9e22-4105-afb9-031ab0f93406 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:57.589518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32346370-9e22-4105-afb9-031ab0f93406 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:57.594512 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed00e17-05ff-4e43-8d49-ec2bca91bf2f None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:57.596998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:57.620021 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32346370-9e22-4105-afb9-031ab0f93406 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:15:57.635532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-32346370-9e22-4105-afb9-031ab0f93406 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:15:57.635532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:57.635532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:57.635532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:57.635532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:57.635532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:57.635532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:57.635532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:57.635532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:57.635532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:57.635532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:57.635532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:57.635532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:15:57.635532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:15:57.635532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:15:57.635532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:15:57.636727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:15:57.636727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:15:57.636727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:15:57.636727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:15:57.636727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:57.636727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:57.636727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:57.636727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:57.636727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:57.636727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:57.636727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:57.636727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:57.636727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:57.636727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:57.636727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:57.636727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:57.638013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:57.638013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:57.638013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:57.638013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:57.638013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:15:57.638013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:15:57.638013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:15:57.638013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:15:57.638013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:15:57.638013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:15:57.638013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:15:57.638013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:57.638013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 409/809] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:15:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:57.645111 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32346370-9e22-4105-afb9-031ab0f93406 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:57.658279 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5fe7a3-8f83-4792-b1fb-5e3a7a0fa4c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:57.659974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5fe7a3-8f83-4792-b1fb-5e3a7a0fa4c4 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:57.660528 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5fe7a3-8f83-4792-b1fb-5e3a7a0fa4c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:57.661031 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5fe7a3-8f83-4792-b1fb-5e3a7a0fa4c4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:57.679254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5fe7a3-8f83-4792-b1fb-5e3a7a0fa4c4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:57.683393 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5fe7a3-8f83-4792-b1fb-5e3a7a0fa4c4 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:57.714487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-ee5fe7a3-8f83-4792-b1fb-5e3a7a0fa4c4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:57.714487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:57.714487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:57.714487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:57.714487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:57.714487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:57.714487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:57.714487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:57.714487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:57.714487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:57.714487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:57.714487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:57.714487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:15:57.714487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:15:57.714487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:15:57.715618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:15:57.715618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:57.715618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:57.715618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:57.715618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:57.715618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:57.715618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:57.715618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:57.715618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:57.715618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:57.715618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:57.715618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:57.715618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:57.715618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:57.715618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:57.715618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:57.715618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:57.716732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:15:57.716732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:15:57.716732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:15:57.716732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:15:57.716732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:15:57.716732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:15:57.716732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:15:57.716732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:57.717582 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 401/810] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:15:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:57.718506 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5fe7a3-8f83-4792-b1fb-5e3a7a0fa4c4 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:57.747873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:57.775407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32346370-9e22-4105-afb9-031ab0f93406 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:57.816355 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0570f4-1b95-4d07-b103-c5aac63dc24d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:57.817069 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0570f4-1b95-4d07-b103-c5aac63dc24d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:57.817349 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0570f4-1b95-4d07-b103-c5aac63dc24d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:57.817649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0570f4-1b95-4d07-b103-c5aac63dc24d None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:57.818279 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0570f4-1b95-4d07-b103-c5aac63dc24d None admin] RBAC: Adding query filter params (name=network) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:57.818585 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0570f4-1b95-4d07-b103-c5aac63dc24d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:57.820007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0570f4-1b95-4d07-b103-c5aac63dc24d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:57.825334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 410/811] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Oct 4 14:15:57 2022] GET /identity/v3/services?name=network => generated 124 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:57.827168 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e6e683-cd5e-4066-9204-7434b71f519f None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:57.827454 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e6e683-cd5e-4066-9204-7434b71f519f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:57.827756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e6e683-cd5e-4066-9204-7434b71f519f None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:57.829298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 411/812] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:57.834634 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e6e683-cd5e-4066-9204-7434b71f519f None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:57.900864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5fe7a3-8f83-4792-b1fb-5e3a7a0fa4c4 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:57.956297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c398d08d-d151-4aae-aa6b-24f5b803c861 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:57.957244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398d08d-d151-4aae-aa6b-24f5b803c861 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:57.957816 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398d08d-d151-4aae-aa6b-24f5b803c861 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:57.958132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398d08d-d151-4aae-aa6b-24f5b803c861 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:57.958983 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c398d08d-d151-4aae-aa6b-24f5b803c861 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:57.959292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c398d08d-d151-4aae-aa6b-24f5b803c861 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:57.960742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c398d08d-d151-4aae-aa6b-24f5b803c861 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:57.975738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 402/813] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:15:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:57.975738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589f5dbd-58d2-4864-81f0-27306619f8a3 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:57.975738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589f5dbd-58d2-4864-81f0-27306619f8a3 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:57.975738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589f5dbd-58d2-4864-81f0-27306619f8a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:57.992114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e6e683-cd5e-4066-9204-7434b71f519f None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:57.992606 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-589f5dbd-58d2-4864-81f0-27306619f8a3 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:57.993638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-589f5dbd-58d2-4864-81f0-27306619f8a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:58.016908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:58.045823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50e7fed-e6a6-49e1-9955-b3c30a6d546d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:58.046840 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50e7fed-e6a6-49e1-9955-b3c30a6d546d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:58.047224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50e7fed-e6a6-49e1-9955-b3c30a6d546d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:58.047644 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50e7fed-e6a6-49e1-9955-b3c30a6d546d None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:58.048734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50e7fed-e6a6-49e1-9955-b3c30a6d546d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:58.049897 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50e7fed-e6a6-49e1-9955-b3c30a6d546d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:58.051491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50e7fed-e6a6-49e1-9955-b3c30a6d546d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:58.077448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 412/814] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:15:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:58.077448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50e7fed-e6a6-49e1-9955-b3c30a6d546d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:58.078823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:58.099535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 403/815] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:57 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:15:58.104516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589f5dbd-58d2-4864-81f0-27306619f8a3 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:58.258499 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50e7fed-e6a6-49e1-9955-b3c30a6d546d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:58.299741 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589f5dbd-58d2-4864-81f0-27306619f8a3 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:58.303960 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6ca75e-43da-4d89-95bb-5cbbc919ec50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:58.305085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6ca75e-43da-4d89-95bb-5cbbc919ec50 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:58.305597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6ca75e-43da-4d89-95bb-5cbbc919ec50 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:58.306122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6ca75e-43da-4d89-95bb-5cbbc919ec50 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:58.306987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6ca75e-43da-4d89-95bb-5cbbc919ec50 None admin] RBAC: Adding query filter params (type=network) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:58.307493 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6ca75e-43da-4d89-95bb-5cbbc919ec50 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:58.309172 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6ca75e-43da-4d89-95bb-5cbbc919ec50 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:58.321076 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 413/816] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Oct 4 14:15:58 2022] GET /identity/v3/services?type=network => generated 350 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:58.322466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6ca75e-43da-4d89-95bb-5cbbc919ec50 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:58.361020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adc5cf8-0c82-4f25-b959-c991a9cc84db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:58.362068 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adc5cf8-0c82-4f25-b959-c991a9cc84db None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:58.362633 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adc5cf8-0c82-4f25-b959-c991a9cc84db None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:58.363160 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adc5cf8-0c82-4f25-b959-c991a9cc84db None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:58.381925 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1adc5cf8-0c82-4f25-b959-c991a9cc84db None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:58.387510 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1adc5cf8-0c82-4f25-b959-c991a9cc84db None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:58.409148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-1adc5cf8-0c82-4f25-b959-c991a9cc84db None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:58.409148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:58.409148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:58.409148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:58.409148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:58.409148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:58.409148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:58.409148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:58.409148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:58.409148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:58.409148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:58.409148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:58.409148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:15:58.409148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:15:58.409148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:15:58.409148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:15:58.410200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:58.410200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:58.410200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:58.410200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:58.410200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:58.410200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:58.410200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:58.410200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:58.410200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:58.410200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:58.410200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:58.410200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:58.410200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:58.410200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:58.410200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:58.410200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:58.410200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:15:58.411146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:15:58.411146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:15:58.411146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:58.411146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 404/817] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:15:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:58.412585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adc5cf8-0c82-4f25-b959-c991a9cc84db None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:58.429686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:58.445216 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:58.497553 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6ca75e-43da-4d89-95bb-5cbbc919ec50 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:58.543049 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753a4263-3c1d-4757-af9a-8d6021e883d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:58.544080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753a4263-3c1d-4757-af9a-8d6021e883d8 None admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:58.544664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753a4263-3c1d-4757-af9a-8d6021e883d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:58.545713 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753a4263-3c1d-4757-af9a-8d6021e883d8 None admin] PATH_INFO: `/v3/projects/895ece8b70ef44adadc3416f81a80731/users/5980e353164c4873bcdd2af847e0a1f1/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:58.548199 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753a4263-3c1d-4757-af9a-8d6021e883d8 None admin] RBAC: Authorizing `identity:create_grant(project_id=895ece8b70ef44adadc3416f81a80731, user_id=5980e353164c4873bcdd2af847e0a1f1, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:58.550011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753a4263-3c1d-4757-af9a-8d6021e883d8 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:58.569093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 414/818] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:15:58 2022] PUT /identity/v3/projects/895ece8b70ef44adadc3416f81a80731/users/5980e353164c4873bcdd2af847e0a1f1/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:15:58.571074 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753a4263-3c1d-4757-af9a-8d6021e883d8 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:58.612009 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adc5cf8-0c82-4f25-b959-c991a9cc84db None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:58.703061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1e25cc-7932-4f5b-a1b4-34ba604a8625 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:58.704163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1e25cc-7932-4f5b-a1b4-34ba604a8625 None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:58.705055 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1e25cc-7932-4f5b-a1b4-34ba604a8625 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:58.705593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1e25cc-7932-4f5b-a1b4-34ba604a8625 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:58.707495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1e25cc-7932-4f5b-a1b4-34ba604a8625 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:58.709722 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1e25cc-7932-4f5b-a1b4-34ba604a8625 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:58.747473 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 405/819] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:15:58 2022] POST /identity/v3/endpoints => generated 337 bytes in 330 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:15:58.747473 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1e25cc-7932-4f5b-a1b4-34ba604a8625 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:58.812523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753a4263-3c1d-4757-af9a-8d6021e883d8 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:58.887751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:58.953066 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2956b89-a550-46a6-a3a6-cd7cff48da81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:58.954083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2956b89-a550-46a6-a3a6-cd7cff48da81 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:58.954531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2956b89-a550-46a6-a3a6-cd7cff48da81 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:58.954976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2956b89-a550-46a6-a3a6-cd7cff48da81 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:58.956145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2956b89-a550-46a6-a3a6-cd7cff48da81 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:58.956984 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2956b89-a550-46a6-a3a6-cd7cff48da81 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:58.957956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:58.959768 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2956b89-a550-46a6-a3a6-cd7cff48da81 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:58.989623 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 415/820] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:15:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 419 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:58.995961 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2956b89-a550-46a6-a3a6-cd7cff48da81 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:59.048925 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1e25cc-7932-4f5b-a1b4-34ba604a8625 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:59.091039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e8fcf4-3cc3-4927-bee0-b6112cd64d61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:59.092782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e8fcf4-3cc3-4927-bee0-b6112cd64d61 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:59.093350 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e8fcf4-3cc3-4927-bee0-b6112cd64d61 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:59.093861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e8fcf4-3cc3-4927-bee0-b6112cd64d61 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:59.094805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e8fcf4-3cc3-4927-bee0-b6112cd64d61 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:59.096680 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e8fcf4-3cc3-4927-bee0-b6112cd64d61 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:59.108504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-b0e8fcf4-3cc3-4927-bee0-b6112cd64d61 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 14:15:59.108504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:59.108504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:15:59.108504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:15:59.108504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:15:59.108504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:15:59.108504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:15:59.108504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:15:59.108504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 14:15:59.108504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:59.108504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:15:59.108504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:59.108504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:59.108504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:59.108504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:59.108504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:59.108504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:59.109753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:59.109753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:59.109753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:59.109753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:59.109753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:59.109753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:59.109753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:15:59.109753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:15:59.109753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:15:59.109753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:15:59.109753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:59.109753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:59.109753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:59.109753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:59.109753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:59.109753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:59.110631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:59.110631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:59.110631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:59.110631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:59.110631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:59.110631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:59.110631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:59.110631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:59.110631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:59.110631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:59.110631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:15:59.110631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:15:59.110631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 14:15:59.110631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:15:59.118570 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 406/821] 10.222.0.64 () {56 vars in 1129 bytes} [Tue Oct 4 14:15:58 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 371 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:15:59.120415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e8fcf4-3cc3-4927-bee0-b6112cd64d61 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:59.222058 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2956b89-a550-46a6-a3a6-cd7cff48da81 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:59.259299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e8fcf4-3cc3-4927-bee0-b6112cd64d61 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:59.283929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8997b33-d055-483c-9439-b4b9fb928d2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:59.285036 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8997b33-d055-483c-9439-b4b9fb928d2a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:59.285419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8997b33-d055-483c-9439-b4b9fb928d2a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:59.285795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8997b33-d055-483c-9439-b4b9fb928d2a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:59.295529 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8997b33-d055-483c-9439-b4b9fb928d2a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:59.297029 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8997b33-d055-483c-9439-b4b9fb928d2a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:59.302064 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:59.314531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-d8997b33-d055-483c-9439-b4b9fb928d2a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:15:59.314531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:15:59.314531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:15:59.314531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:15:59.314531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:15:59.314531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:15:59.314531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:15:59.314531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:15:59.314531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:15:59.314531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:15:59.314531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:15:59.314531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:15:59.314531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:15:59.314531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:15:59.314531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:15:59.314531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:15:59.315598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:15:59.317001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:15:59.317001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:15:59.317001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:15:59.317001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:15:59.317001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:15:59.317001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:15:59.317001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:15:59.317001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:15:59.317001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:15:59.317001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:15:59.317001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 416/822] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Oct 4 14:15:58 2022] GET /identity/v3/users/nova => generated 82 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:15:59.322384 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a179e819-b5fd-433d-a190-2ac1644b9dc8 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:59.322682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83f18b7-2032-418b-9f69-c57f7bd691bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:59.323210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a179e819-b5fd-433d-a190-2ac1644b9dc8 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:59.323399 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83f18b7-2032-418b-9f69-c57f7bd691bb None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:59.323557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a179e819-b5fd-433d-a190-2ac1644b9dc8 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:59.323690 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83f18b7-2032-418b-9f69-c57f7bd691bb None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:59.323815 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83f18b7-2032-418b-9f69-c57f7bd691bb None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:59.324442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83f18b7-2032-418b-9f69-c57f7bd691bb None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:59.324744 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83f18b7-2032-418b-9f69-c57f7bd691bb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:59.325213 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 417/823] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:15:59 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:15:59.325916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a179e819-b5fd-433d-a190-2ac1644b9dc8 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:59.326697 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83f18b7-2032-418b-9f69-c57f7bd691bb None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:59.332916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 407/824] 10.222.0.64 () {56 vars in 1124 bytes} [Tue Oct 4 14:15:59 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:15:59.334330 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b69a0e2-9bee-41c0-b438-2cffa10b3780 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:59.334608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b69a0e2-9bee-41c0-b438-2cffa10b3780 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:59.334888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b69a0e2-9bee-41c0-b438-2cffa10b3780 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:59.355817 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-9b69a0e2-9bee-41c0-b438-2cffa10b3780 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:15:59.357122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9b69a0e2-9bee-41c0-b438-2cffa10b3780 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:15:59.392906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:59.465038 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 408/825] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:15:59 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:15:59.465038 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b69a0e2-9bee-41c0-b438-2cffa10b3780 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:59.484587 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a179e819-b5fd-433d-a190-2ac1644b9dc8 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:59.538702 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a419ec-40e8-43a7-a942-ace4f4706a86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:59.539848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a419ec-40e8-43a7-a942-ace4f4706a86 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:59.540427 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a419ec-40e8-43a7-a942-ace4f4706a86 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:59.541313 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a419ec-40e8-43a7-a942-ace4f4706a86 None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:59.542421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a419ec-40e8-43a7-a942-ace4f4706a86 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:59.542952 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a419ec-40e8-43a7-a942-ace4f4706a86 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:59.547025 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a419ec-40e8-43a7-a942-ace4f4706a86 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:59.600381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 418/826] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Oct 4 14:15:59 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:59.602087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a419ec-40e8-43a7-a942-ace4f4706a86 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:59.620408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b69a0e2-9bee-41c0-b438-2cffa10b3780 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:59.664135 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:59.694263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2db869-66bd-4ecf-ba9f-890cc1f6aae6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:59.695738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2db869-66bd-4ecf-ba9f-890cc1f6aae6 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:59.696453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2db869-66bd-4ecf-ba9f-890cc1f6aae6 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:59.697106 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2db869-66bd-4ecf-ba9f-890cc1f6aae6 None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:59.698444 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2db869-66bd-4ecf-ba9f-890cc1f6aae6 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:15:59.699162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2db869-66bd-4ecf-ba9f-890cc1f6aae6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:59.702606 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2db869-66bd-4ecf-ba9f-890cc1f6aae6 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:59.712871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 409/827] 10.222.0.64 () {56 vars in 1124 bytes} [Tue Oct 4 14:15:59 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:15:59.718795 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2db869-66bd-4ecf-ba9f-890cc1f6aae6 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:59.811864 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:15:59.817239 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a419ec-40e8-43a7-a942-ace4f4706a86 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:59.869127 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfb9bf2-a42e-46a5-ab1d-4c3c743aeb19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:59.869875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfb9bf2-a42e-46a5-ab1d-4c3c743aeb19 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:59.870234 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfb9bf2-a42e-46a5-ab1d-4c3c743aeb19 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:59.870731 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfb9bf2-a42e-46a5-ab1d-4c3c743aeb19 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:59.876798 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2db869-66bd-4ecf-ba9f-890cc1f6aae6 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:15:59.925334 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a36bcd-340e-457b-8800-04010ad8a794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:15:59.926093 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a36bcd-340e-457b-8800-04010ad8a794 None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:15:59.926457 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a36bcd-340e-457b-8800-04010ad8a794 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:15:59.926815 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a36bcd-340e-457b-8800-04010ad8a794 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:15:59.927842 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a36bcd-340e-457b-8800-04010ad8a794 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:59.929484 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a36bcd-340e-457b-8800-04010ad8a794 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:15:59.968343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 410/828] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:15:59 2022] POST /identity/v3/endpoints => generated 346 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:15:59.968343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a36bcd-340e-457b-8800-04010ad8a794 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:15:59.984935 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfb9bf2-a42e-46a5-ab1d-4c3c743aeb19 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:15:59.987787 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfb9bf2-a42e-46a5-ab1d-4c3c743aeb19 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:00.050817 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:00.176416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a36bcd-340e-457b-8800-04010ad8a794 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:00.196119 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:00.204085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 419/829] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:15:59 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 603 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 14:16:00.211429 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfb9bf2-a42e-46a5-ab1d-4c3c743aeb19 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:00.253126 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e623cd-6e33-4b21-bde2-bc35d3e022e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:00.254181 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e623cd-6e33-4b21-bde2-bc35d3e022e9 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:00.254466 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e623cd-6e33-4b21-bde2-bc35d3e022e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:00.255320 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e623cd-6e33-4b21-bde2-bc35d3e022e9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:00.270562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e623cd-6e33-4b21-bde2-bc35d3e022e9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:16:00.275738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-e1e623cd-6e33-4b21-bde2-bc35d3e022e9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:16:00.275738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:00.275738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:00.275738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:00.275738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:00.275738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:00.275738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:00.275738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:00.275738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:00.275738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:00.275738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:00.275738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:00.275738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:16:00.275738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:16:00.275738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:16:00.275738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:16:00.276806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:16:00.276806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:16:00.276806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:16:00.276806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:16:00.276806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:00.276806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:00.276806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:00.276806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:00.276806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:00.276806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:00.276806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:00.276806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:00.276806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:00.276806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:00.276806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:00.276806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:00.277922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:00.277922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:00.277922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:00.277922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:00.277922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:16:00.277922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:16:00.277922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:16:00.277922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:16:00.277922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:16:00.277922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:16:00.277922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:16:00.277922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:00.277922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 411/830] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:15:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:00.286526 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e623cd-6e33-4b21-bde2-bc35d3e022e9 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:00.405925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfb9bf2-a42e-46a5-ab1d-4c3c743aeb19 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:00.450713 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449cf3db-5089-44e2-b513-2c1cea1e3939 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:00.451994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449cf3db-5089-44e2-b513-2c1cea1e3939 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:00.452639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449cf3db-5089-44e2-b513-2c1cea1e3939 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:00.452639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449cf3db-5089-44e2-b513-2c1cea1e3939 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:00.467904 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449cf3db-5089-44e2-b513-2c1cea1e3939 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:00.469888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449cf3db-5089-44e2-b513-2c1cea1e3939 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:00.476567 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:00.490346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e623cd-6e33-4b21-bde2-bc35d3e022e9 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:00.493448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-449cf3db-5089-44e2-b513-2c1cea1e3939 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:00.493448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:00.493448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:00.493448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:00.493448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:00.493448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:00.493448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:00.493448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:00.493448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:00.493448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:00.493448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:00.493448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:00.493448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:00.493448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:00.493448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:00.493448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:00.494653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:00.494653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:00.494653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:00.494653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:00.494653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:00.494653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:00.494653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:00.494653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:00.494653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:00.494653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:00.494653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:00.494653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:00.494653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:00.494653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:00.494653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:00.494653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:00.494653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:00.495755 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:00.495755 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:00.495755 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:00.499049 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 420/831] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:00.507106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449cf3db-5089-44e2-b513-2c1cea1e3939 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:00.564165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bcda03-ddd2-475c-910a-6fc4f35f4d60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:00.564821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bcda03-ddd2-475c-910a-6fc4f35f4d60 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:00.565098 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bcda03-ddd2-475c-910a-6fc4f35f4d60 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:00.565818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bcda03-ddd2-475c-910a-6fc4f35f4d60 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:00.566147 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bcda03-ddd2-475c-910a-6fc4f35f4d60 None admin] RBAC: Adding query filter params (name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:00.566451 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bcda03-ddd2-475c-910a-6fc4f35f4d60 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:00.568177 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bcda03-ddd2-475c-910a-6fc4f35f4d60 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:00.583865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 412/832] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:16:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:00.585824 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bcda03-ddd2-475c-910a-6fc4f35f4d60 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:00.585985 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:00.648036 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449cf3db-5089-44e2-b513-2c1cea1e3939 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:00.699537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e450804-f891-4c27-87aa-f4c102e68010 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:00.700841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e450804-f891-4c27-87aa-f4c102e68010 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:00.701873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e450804-f891-4c27-87aa-f4c102e68010 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:00.702354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e450804-f891-4c27-87aa-f4c102e68010 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:00.703522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e450804-f891-4c27-87aa-f4c102e68010 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:00.704563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e450804-f891-4c27-87aa-f4c102e68010 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:00.706200 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e450804-f891-4c27-87aa-f4c102e68010 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:00.722275 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bcda03-ddd2-475c-910a-6fc4f35f4d60 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:00.728674 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 421/833] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:00.730521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e450804-f891-4c27-87aa-f4c102e68010 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:00.777150 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421d8055-d8a1-4614-9233-a0235e5b0c7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:00.779011 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421d8055-d8a1-4614-9233-a0235e5b0c7c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:00.780531 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421d8055-d8a1-4614-9233-a0235e5b0c7c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:00.781196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421d8055-d8a1-4614-9233-a0235e5b0c7c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:00.808077 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421d8055-d8a1-4614-9233-a0235e5b0c7c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:00.812102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421d8055-d8a1-4614-9233-a0235e5b0c7c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:00.852304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-421d8055-d8a1-4614-9233-a0235e5b0c7c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:00.852304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:00.852304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:00.852304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:00.852304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:00.852304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:00.852304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:00.852304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:00.852304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:00.852304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:00.852304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:00.852304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:00.852304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:00.852304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:00.852304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:00.852304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:00.854450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:00.854450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:00.854450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:00.854450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:00.854450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:00.854450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:00.854450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:00.854450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:00.854450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:00.854450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:00.854450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:00.854450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:00.854450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:00.854450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:00.854450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:00.854450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:00.854450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:00.856862 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:00.856862 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:00.856862 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:00.864558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 413/834] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:00.868047 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421d8055-d8a1-4614-9233-a0235e5b0c7c None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:00.876116 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e450804-f891-4c27-87aa-f4c102e68010 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:00.886806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:00.896951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:00.934616 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11d680a-e093-442d-b364-9e946c0e77d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:00.935474 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11d680a-e093-442d-b364-9e946c0e77d6 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:00.937614 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11d680a-e093-442d-b364-9e946c0e77d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:00.938761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11d680a-e093-442d-b364-9e946c0e77d6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:00.953621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11d680a-e093-442d-b364-9e946c0e77d6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:00.955254 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11d680a-e093-442d-b364-9e946c0e77d6 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:00.971982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-c11d680a-e093-442d-b364-9e946c0e77d6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:00.971982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:00.971982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:00.971982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:00.971982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:00.971982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:00.971982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:00.971982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:00.971982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:00.971982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:00.971982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:00.971982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:00.971982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:16:00.971982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:16:00.971982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:16:00.973886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:16:00.973886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:00.973886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:00.973886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:00.973886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:00.973886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:00.973886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:00.973886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:00.973886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:00.973886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:00.973886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:00.973886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:00.973886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:00.973886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:00.973886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:00.973886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:00.973886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:00.975645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:16:00.975645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:16:00.975645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:16:00.975645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:16:00.975645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:16:00.975645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:16:00.975645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:00.975645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:00.980475 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 422/835] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:16:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:00.982005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11d680a-e093-442d-b364-9e946c0e77d6 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:01.067930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421d8055-d8a1-4614-9233-a0235e5b0c7c None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:01.129643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c316d4e-4dc7-4553-ba2f-4be46a40aaca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:01.131628 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c316d4e-4dc7-4553-ba2f-4be46a40aaca None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:01.132631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c316d4e-4dc7-4553-ba2f-4be46a40aaca None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:01.133296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c316d4e-4dc7-4553-ba2f-4be46a40aaca None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:01.135019 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c316d4e-4dc7-4553-ba2f-4be46a40aaca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:01.136533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c316d4e-4dc7-4553-ba2f-4be46a40aaca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:01.141084 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c316d4e-4dc7-4553-ba2f-4be46a40aaca None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:01.160519 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11d680a-e093-442d-b364-9e946c0e77d6 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:01.181455 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 414/836] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:01.187240 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c316d4e-4dc7-4553-ba2f-4be46a40aaca None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:01.240694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdee357-412e-4c3b-ab18-7ee50d69e98b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:01.241230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdee357-412e-4c3b-ab18-7ee50d69e98b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:01.241513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdee357-412e-4c3b-ab18-7ee50d69e98b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:01.241790 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdee357-412e-4c3b-ab18-7ee50d69e98b None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:01.242585 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fdee357-412e-4c3b-ab18-7ee50d69e98b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:01.242892 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fdee357-412e-4c3b-ab18-7ee50d69e98b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:01.244439 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fdee357-412e-4c3b-ab18-7ee50d69e98b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:01.276622 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 423/837] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:16:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:01.285686 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdee357-412e-4c3b-ab18-7ee50d69e98b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:01.309861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:01.351391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c316d4e-4dc7-4553-ba2f-4be46a40aaca None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:01.353444 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:01.400633 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a2a701-30fa-4acb-927f-69f39b4d68b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:01.401290 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a2a701-30fa-4acb-927f-69f39b4d68b6 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:01.401578 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a2a701-30fa-4acb-927f-69f39b4d68b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:01.401852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a2a701-30fa-4acb-927f-69f39b4d68b6 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:01.411970 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a2a701-30fa-4acb-927f-69f39b4d68b6 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:01.413564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a2a701-30fa-4acb-927f-69f39b4d68b6 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:01.427397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-f2a2a701-30fa-4acb-927f-69f39b4d68b6 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 14:16:01.427397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:01.427397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:01.427397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:01.427397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:01.427397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:01.427397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:01.427397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:01.427397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:01.427397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:01.427397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:01.427397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:01.427397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:16:01.427397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:16:01.427397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:16:01.427397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:01.428710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:01.429665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:01.429665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:01.429665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:16:01.429665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:16:01.429665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:16:01.429665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:16:01.429665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:16:01.429665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:16:01.429665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 14:16:01.429665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:01.429665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 415/838] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Oct 4 14:16:01 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:01.431347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98356286-9c6b-45af-8df7-d82db991c49e None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:01.431347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98356286-9c6b-45af-8df7-d82db991c49e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:01.431347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98356286-9c6b-45af-8df7-d82db991c49e None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:01.434749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 416/839] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:01 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:01.438229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380690e6-fb38-430d-9d21-dc92dbb1f169 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:01.438515 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380690e6-fb38-430d-9d21-dc92dbb1f169 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:01.438790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380690e6-fb38-430d-9d21-dc92dbb1f169 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:01.458652 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-380690e6-fb38-430d-9d21-dc92dbb1f169 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:01.460359 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-380690e6-fb38-430d-9d21-dc92dbb1f169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:01.482616 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdee357-412e-4c3b-ab18-7ee50d69e98b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:01.528692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed43967-6231-4ec0-b67d-220931cec210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:01.530384 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed43967-6231-4ec0-b67d-220931cec210 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:01.531002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed43967-6231-4ec0-b67d-220931cec210 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:01.531475 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed43967-6231-4ec0-b67d-220931cec210 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:01.533065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed43967-6231-4ec0-b67d-220931cec210 None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=03081d7d581745c88c12e9d089fcb2d3) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:01.533413 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed43967-6231-4ec0-b67d-220931cec210 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:01.536143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed43967-6231-4ec0-b67d-220931cec210 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:01.552355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 417/840] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:01 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:16:01.552355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380690e6-fb38-430d-9d21-dc92dbb1f169 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:01.552659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 424/841] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:16:01 2022] GET /identity/v3/role_assignments?user.id=03081d7d581745c88c12e9d089fcb2d3&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 608 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:01.680276 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:01.759093 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380690e6-fb38-430d-9d21-dc92dbb1f169 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:01.811157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec69aa6-6b7b-4fe2-9ccb-a616d3deeb4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:01.811717 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec69aa6-6b7b-4fe2-9ccb-a616d3deeb4b None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:01.811998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec69aa6-6b7b-4fe2-9ccb-a616d3deeb4b None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:01.812274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec69aa6-6b7b-4fe2-9ccb-a616d3deeb4b None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:01.813366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec69aa6-6b7b-4fe2-9ccb-a616d3deeb4b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:01.813676 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec69aa6-6b7b-4fe2-9ccb-a616d3deeb4b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:01.815192 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec69aa6-6b7b-4fe2-9ccb-a616d3deeb4b None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:01.838053 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 418/842] 10.222.0.64 () {56 vars in 1149 bytes} [Tue Oct 4 14:16:01 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:01.844988 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed43967-6231-4ec0-b67d-220931cec210 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:01.960182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:01.962857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec69aa6-6b7b-4fe2-9ccb-a616d3deeb4b None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:01.967220 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed43967-6231-4ec0-b67d-220931cec210 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:02.007937 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce80f642-f54e-4ab2-8165-cef7e211ba6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:02.008600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80f642-f54e-4ab2-8165-cef7e211ba6c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:02.008890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80f642-f54e-4ab2-8165-cef7e211ba6c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:02.009164 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80f642-f54e-4ab2-8165-cef7e211ba6c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:02.132686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec69aa6-6b7b-4fe2-9ccb-a616d3deeb4b None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:02.145371 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce80f642-f54e-4ab2-8165-cef7e211ba6c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:02.150372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce80f642-f54e-4ab2-8165-cef7e211ba6c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:02.160071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:02.185708 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9ba68c-17df-4e30-be3a-8667d5d6651d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:02.186338 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9ba68c-17df-4e30-be3a-8667d5d6651d None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:02.186603 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9ba68c-17df-4e30-be3a-8667d5d6651d None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:02.186878 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9ba68c-17df-4e30-be3a-8667d5d6651d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:02.201561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9ba68c-17df-4e30-be3a-8667d5d6651d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:16:02.207899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-1b9ba68c-17df-4e30-be3a-8667d5d6651d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:16:02.207899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:02.207899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:02.207899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:02.207899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:02.207899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:02.207899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:02.207899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:02.207899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:02.207899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:02.207899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:02.207899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:02.207899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:16:02.207899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:16:02.207899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:16:02.207899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:16:02.212871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:16:02.212871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:16:02.212871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:16:02.212871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:16:02.212871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:02.212871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:02.212871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:02.212871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:02.212871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:02.212871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:02.212871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:02.212871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:02.212871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:02.212871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:02.212871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:02.212871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:02.213778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:02.213778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:02.213778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:02.213778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:02.213778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:16:02.213778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:16:02.213778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:16:02.213778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:16:02.213778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:16:02.213778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:16:02.213778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:16:02.213778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:02.213778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 419/843] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:16:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:02.219577 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9ba68c-17df-4e30-be3a-8667d5d6651d None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:02.327467 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:02.344296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 425/844] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:16:01 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 500 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 14:16:02.353434 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce80f642-f54e-4ab2-8165-cef7e211ba6c None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:02.410130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9ba68c-17df-4e30-be3a-8667d5d6651d None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:02.482376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2755e18-a92f-4ce5-acad-f3babd152e54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:02.483771 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2755e18-a92f-4ce5-acad-f3babd152e54 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:02.484528 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2755e18-a92f-4ce5-acad-f3babd152e54 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:02.485206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2755e18-a92f-4ce5-acad-f3babd152e54 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:02.487020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2755e18-a92f-4ce5-acad-f3babd152e54 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:02.487865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2755e18-a92f-4ce5-acad-f3babd152e54 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:02.490788 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2755e18-a92f-4ce5-acad-f3babd152e54 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:02.515966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 420/845] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:16:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:02.517930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2755e18-a92f-4ce5-acad-f3babd152e54 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:02.557727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce80f642-f54e-4ab2-8165-cef7e211ba6c None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:02.611362 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:02.639302 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44dfceae-77e3-483f-9cc7-51686603ce15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:02.640419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dfceae-77e3-483f-9cc7-51686603ce15 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:02.640844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dfceae-77e3-483f-9cc7-51686603ce15 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:02.641275 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dfceae-77e3-483f-9cc7-51686603ce15 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:02.663264 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44dfceae-77e3-483f-9cc7-51686603ce15 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:02.665739 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44dfceae-77e3-483f-9cc7-51686603ce15 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:02.691835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-44dfceae-77e3-483f-9cc7-51686603ce15 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:02.691835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:02.691835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:02.691835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:02.691835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:02.691835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:02.691835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:02.691835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:02.691835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:02.691835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:02.691835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:02.691835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:02.691835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:02.691835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:02.691835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:02.691835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:02.693543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:02.693543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:02.693543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:02.693543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:02.693543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:02.693543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:02.693543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:02.693543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:02.693543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:02.693543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:02.693543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:02.693543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:02.693543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:02.693543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:02.693543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:02.693543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:02.693543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:02.695096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:02.695096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:02.695096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:02.701115 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 426/846] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 348 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:02.703275 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c8cbc2-914c-4438-82d2-d4226e1b76a7 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:02.703749 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c8cbc2-914c-4438-82d2-d4226e1b76a7 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:02.704186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c8cbc2-914c-4438-82d2-d4226e1b76a7 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:02.712198 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 427/847] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:02.712198 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c8cbc2-914c-4438-82d2-d4226e1b76a7 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:02.722629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2755e18-a92f-4ce5-acad-f3babd152e54 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:02.769979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee53d92-761c-410a-ae50-87c221ff4db7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:02.771076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee53d92-761c-410a-ae50-87c221ff4db7 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:02.771510 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee53d92-761c-410a-ae50-87c221ff4db7 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:02.771998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee53d92-761c-410a-ae50-87c221ff4db7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:02.786682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee53d92-761c-410a-ae50-87c221ff4db7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:02.788671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee53d92-761c-410a-ae50-87c221ff4db7 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:02.804579 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:02.816958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-0ee53d92-761c-410a-ae50-87c221ff4db7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:02.816958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:02.816958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:02.816958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:02.816958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:02.816958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:02.816958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:02.816958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:02.816958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:02.816958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:02.816958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:02.816958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:02.816958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:02.816958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:02.816958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:02.816958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:02.818796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:02.818796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:02.818796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:02.818796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:02.818796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:02.818796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:02.818796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:02.818796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:02.818796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:02.818796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:02.818796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:02.818796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:02.818796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:02.818796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:02.818796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:02.818796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:02.818796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:02.820191 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:02.820191 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:02.820191 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:02.828706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 421/848] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:02.828706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13744d73-3069-464a-a366-5fbee2e4dab0 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:02.828706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13744d73-3069-464a-a366-5fbee2e4dab0 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:02.828706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13744d73-3069-464a-a366-5fbee2e4dab0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:02.866065 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-13744d73-3069-464a-a366-5fbee2e4dab0 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:02.869238 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-13744d73-3069-464a-a366-5fbee2e4dab0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:02.882955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c8cbc2-914c-4438-82d2-d4226e1b76a7 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:02.937595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc71fcc0-6bc2-4afd-b2f6-3ead1c113995 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:02.938162 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc71fcc0-6bc2-4afd-b2f6-3ead1c113995 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:02.938461 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc71fcc0-6bc2-4afd-b2f6-3ead1c113995 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:02.938823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc71fcc0-6bc2-4afd-b2f6-3ead1c113995 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:02.939543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc71fcc0-6bc2-4afd-b2f6-3ead1c113995 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:02.939867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc71fcc0-6bc2-4afd-b2f6-3ead1c113995 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:02.941433 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc71fcc0-6bc2-4afd-b2f6-3ead1c113995 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:02.964258 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 428/849] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:02.968042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc71fcc0-6bc2-4afd-b2f6-3ead1c113995 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:02.983482 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 422/850] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:02 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:02.984521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13744d73-3069-464a-a366-5fbee2e4dab0 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:03.015630 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:03.122911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13744d73-3069-464a-a366-5fbee2e4dab0 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:03.173318 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c33e7b-30b0-44f8-a088-34f2b160af12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:03.174392 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c33e7b-30b0-44f8-a088-34f2b160af12 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:03.174392 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c33e7b-30b0-44f8-a088-34f2b160af12 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:03.174654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c33e7b-30b0-44f8-a088-34f2b160af12 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:03.182482 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c33e7b-30b0-44f8-a088-34f2b160af12 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:03.183952 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c33e7b-30b0-44f8-a088-34f2b160af12 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:03.198092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-59c33e7b-30b0-44f8-a088-34f2b160af12 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:03.198092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:03.198092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:03.198092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:03.198092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:03.198092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:03.198092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:03.198092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:03.198092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:03.198092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:03.198092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:03.198092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:03.198092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:16:03.198092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:16:03.198092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:16:03.199894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:16:03.199894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:03.199894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:03.199894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:03.199894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:03.199894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:03.199894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:03.199894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:03.199894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:03.199894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:03.199894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:03.199894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:03.199894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:03.199894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:03.199894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:03.199894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:03.199894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:03.202622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:16:03.202622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:16:03.202622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:16:03.202622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:16:03.202622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:16:03.202622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:16:03.202622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:03.202622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:03.202622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 423/851] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:16:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:03.208449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c33e7b-30b0-44f8-a088-34f2b160af12 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:03.215577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc71fcc0-6bc2-4afd-b2f6-3ead1c113995 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:03.226449 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:03.257588 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1856d5db-84a8-4837-9bec-aedd2cd621bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:03.258289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1856d5db-84a8-4837-9bec-aedd2cd621bd None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:03.258564 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1856d5db-84a8-4837-9bec-aedd2cd621bd None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:03.258833 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1856d5db-84a8-4837-9bec-aedd2cd621bd None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:03.259570 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1856d5db-84a8-4837-9bec-aedd2cd621bd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:03.259922 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1856d5db-84a8-4837-9bec-aedd2cd621bd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:03.261339 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1856d5db-84a8-4837-9bec-aedd2cd621bd None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:03.280981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 429/852] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:03.282028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1856d5db-84a8-4837-9bec-aedd2cd621bd None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:03.388990 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:03.412947 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c33e7b-30b0-44f8-a088-34f2b160af12 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:03.446776 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1856d5db-84a8-4837-9bec-aedd2cd621bd None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:03.453304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0563748-57fe-48f1-9ef8-90c2a49d07d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:03.453929 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0563748-57fe-48f1-9ef8-90c2a49d07d4 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:03.454291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0563748-57fe-48f1-9ef8-90c2a49d07d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:03.454627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0563748-57fe-48f1-9ef8-90c2a49d07d4 None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:03.455411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0563748-57fe-48f1-9ef8-90c2a49d07d4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:03.455785 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0563748-57fe-48f1-9ef8-90c2a49d07d4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:03.458133 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0563748-57fe-48f1-9ef8-90c2a49d07d4 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:03.476551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 424/853] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:16:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:03.483644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0563748-57fe-48f1-9ef8-90c2a49d07d4 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:03.515086 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9eb33a-9787-4c01-b430-8b0416caf880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:03.516017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9eb33a-9787-4c01-b430-8b0416caf880 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:03.516414 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9eb33a-9787-4c01-b430-8b0416caf880 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:03.516784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9eb33a-9787-4c01-b430-8b0416caf880 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:03.527429 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9eb33a-9787-4c01-b430-8b0416caf880 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:03.529722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9eb33a-9787-4c01-b430-8b0416caf880 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:03.544118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-5a9eb33a-9787-4c01-b430-8b0416caf880 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 14:16:03.544118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:03.544118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:03.544118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:03.544118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:03.544118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:03.544118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:03.544118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:03.544118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:03.544118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:03.544118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:03.544118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:03.544118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:16:03.544118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:16:03.544118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:16:03.544118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:03.546517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:03.548411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:03.548411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:03.548411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:16:03.548411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:16:03.548411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:16:03.548411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:16:03.548411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:16:03.548411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:16:03.548411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 14:16:03.548411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:03.549481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 430/854] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:16:03 2022] GET /identity/v3/users/swift => generated 83 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:03.550989 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9eb33a-9787-4c01-b430-8b0416caf880 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:03.612405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:03.657760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0563748-57fe-48f1-9ef8-90c2a49d07d4 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:03.704251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15377cf-86ae-43f0-bc45-21653bfacbb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:03.705014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15377cf-86ae-43f0-bc45-21653bfacbb4 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:03.705413 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15377cf-86ae-43f0-bc45-21653bfacbb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:03.705818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15377cf-86ae-43f0-bc45-21653bfacbb4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:03.706675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15377cf-86ae-43f0-bc45-21653bfacbb4 None admin] RBAC: Adding query filter params (role.id=3ebe3c0be8c34c5eae533311e9b44176, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=34afa4aac10c4a018ca05a0ba34c940a) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:03.707105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15377cf-86ae-43f0-bc45-21653bfacbb4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:03.708743 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15377cf-86ae-43f0-bc45-21653bfacbb4 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:03.717353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 425/855] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:16:03 2022] GET /identity/v3/role_assignments?user.id=34afa4aac10c4a018ca05a0ba34c940a&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=3ebe3c0be8c34c5eae533311e9b44176 => generated 608 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:03.718449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15377cf-86ae-43f0-bc45-21653bfacbb4 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:03.736501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:03.776457 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9eb33a-9787-4c01-b430-8b0416caf880 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:03.852190 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d19d8f3-c1cd-4f05-87ef-1c033a62ca7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:03.853863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d19d8f3-c1cd-4f05-87ef-1c033a62ca7c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:03.855137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d19d8f3-c1cd-4f05-87ef-1c033a62ca7c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:03.857616 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d19d8f3-c1cd-4f05-87ef-1c033a62ca7c None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:03.858022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d19d8f3-c1cd-4f05-87ef-1c033a62ca7c None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:03.860561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d19d8f3-c1cd-4f05-87ef-1c033a62ca7c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:03.877137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-4d19d8f3-c1cd-4f05-87ef-1c033a62ca7c None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 14:16:03.877137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:03.877137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:16:03.877137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:16:03.877137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:16:03.877137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:16:03.877137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:16:03.877137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:03.877137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 14:16:03.877137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:03.877137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:03.877137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:03.877137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:03.877137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:03.877137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:03.877137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:03.877137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:03.879101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:03.879101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:03.879101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:03.879101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:03.879101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:03.879101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:03.879101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:16:03.879101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:16:03.879101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:16:03.879101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:16:03.879101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:03.879101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:03.879101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:03.879101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:03.879101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:03.879101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:03.880773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:03.880773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:03.880773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:03.880773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:03.880773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:03.880773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:03.880773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:03.880773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:03.880773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:03.880773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:03.880773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:16:03.880773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:03.880773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 14:16:03.880773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:03.885022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 431/856] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:16:03 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 335 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:03.889945 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d19d8f3-c1cd-4f05-87ef-1c033a62ca7c None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:03.936599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15377cf-86ae-43f0-bc45-21653bfacbb4 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:04.002441 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e994faa2-2bb6-40b9-9903-080b1e37b3cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:04.003333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e994faa2-2bb6-40b9-9903-080b1e37b3cd None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:04.003877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e994faa2-2bb6-40b9-9903-080b1e37b3cd None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:04.004495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e994faa2-2bb6-40b9-9903-080b1e37b3cd None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:04.005748 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e994faa2-2bb6-40b9-9903-080b1e37b3cd None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:04.006249 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e994faa2-2bb6-40b9-9903-080b1e37b3cd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:04.008879 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e994faa2-2bb6-40b9-9903-080b1e37b3cd None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:04.046938 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 426/857] 10.222.0.64 () {56 vars in 1135 bytes} [Tue Oct 4 14:16:03 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 329 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:04.050061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67665ce4-0875-4169-82ce-e8a771031280 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:04.050536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67665ce4-0875-4169-82ce-e8a771031280 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:04.050995 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67665ce4-0875-4169-82ce-e8a771031280 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:04.053488 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 427/858] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:04 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:16:04.057524 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67665ce4-0875-4169-82ce-e8a771031280 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:04.104185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:04.125725 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d19d8f3-c1cd-4f05-87ef-1c033a62ca7c None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:04.164754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:04.193249 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e896ae-fb42-4b07-b769-be4037a5cbdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:04.194260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e896ae-fb42-4b07-b769-be4037a5cbdd None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:04.194712 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e896ae-fb42-4b07-b769-be4037a5cbdd None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:04.195182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e896ae-fb42-4b07-b769-be4037a5cbdd None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:04.196279 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e896ae-fb42-4b07-b769-be4037a5cbdd None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:04.196795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e896ae-fb42-4b07-b769-be4037a5cbdd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:04.199331 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e896ae-fb42-4b07-b769-be4037a5cbdd None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:04.208298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 432/859] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:16:03 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:04.210650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e62018-6d06-41e5-8c88-59b27d3cecbb None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:04.211114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e62018-6d06-41e5-8c88-59b27d3cecbb None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:04.211582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e62018-6d06-41e5-8c88-59b27d3cecbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:04.241661 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-b9e62018-6d06-41e5-8c88-59b27d3cecbb None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:04.244851 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b9e62018-6d06-41e5-8c88-59b27d3cecbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:04.247283 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67665ce4-0875-4169-82ce-e8a771031280 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:04.295825 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d7ed53-b321-4d6e-9f0f-a1d1c3814e96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:04.296618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d7ed53-b321-4d6e-9f0f-a1d1c3814e96 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:04.297026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d7ed53-b321-4d6e-9f0f-a1d1c3814e96 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:04.297487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d7ed53-b321-4d6e-9f0f-a1d1c3814e96 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:04.390822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 433/860] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:04 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:04.390822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e62018-6d06-41e5-8c88-59b27d3cecbb None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:04.433049 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d7ed53-b321-4d6e-9f0f-a1d1c3814e96 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:04.437288 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d7ed53-b321-4d6e-9f0f-a1d1c3814e96 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:04.556557 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:04.585071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e62018-6d06-41e5-8c88-59b27d3cecbb None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:04.588850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:04.633807 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 428/861] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:16:04 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 577 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 04 14:16:04.641879 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d7ed53-b321-4d6e-9f0f-a1d1c3814e96 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:04.648517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e4fd26-22cc-45a1-a465-369600f93280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:04.650430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e4fd26-22cc-45a1-a465-369600f93280 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:04.651146 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e4fd26-22cc-45a1-a465-369600f93280 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:04.651836 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e4fd26-22cc-45a1-a465-369600f93280 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:04.653334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e4fd26-22cc-45a1-a465-369600f93280 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:04.654917 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e4fd26-22cc-45a1-a465-369600f93280 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:04.657488 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e4fd26-22cc-45a1-a465-369600f93280 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:04.666304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 434/862] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:16:04 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:04.803120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d7ed53-b321-4d6e-9f0f-a1d1c3814e96 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:04.847559 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0014c769-5590-4fd5-a528-4f41d3f4731c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:04.848111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0014c769-5590-4fd5-a528-4f41d3f4731c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:04.848471 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0014c769-5590-4fd5-a528-4f41d3f4731c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:04.848954 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0014c769-5590-4fd5-a528-4f41d3f4731c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:04.863225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0014c769-5590-4fd5-a528-4f41d3f4731c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:04.864756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0014c769-5590-4fd5-a528-4f41d3f4731c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:04.885271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-0014c769-5590-4fd5-a528-4f41d3f4731c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:04.885271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:04.885271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:04.885271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:04.885271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:04.885271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:04.885271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:04.885271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:04.885271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:04.885271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:04.885271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:04.885271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:04.885271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:04.885271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:04.885271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:04.885271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:04.887085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:04.887085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:04.887085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:04.887085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:04.887085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:04.887085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:04.887085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:04.887085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:04.887085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:04.887085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:04.887085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:04.887085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:04.887085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:04.887085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:04.887085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:04.887085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:04.887085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:04.890722 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:04.890722 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:04.890722 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:04.890722 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 429/863] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:04.896188 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e4fd26-22cc-45a1-a465-369600f93280 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:04.930177 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0014c769-5590-4fd5-a528-4f41d3f4731c None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:04.964727 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:05.134017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0014c769-5590-4fd5-a528-4f41d3f4731c None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:05.138302 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e4fd26-22cc-45a1-a465-369600f93280 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:05.204339 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2486259-8311-4ac7-8e24-4c3482bbe448 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:05.206006 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2486259-8311-4ac7-8e24-4c3482bbe448 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:05.206809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2486259-8311-4ac7-8e24-4c3482bbe448 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:05.207577 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2486259-8311-4ac7-8e24-4c3482bbe448 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:05.208978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2486259-8311-4ac7-8e24-4c3482bbe448 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:05.211490 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeaf69c-81f1-4a36-9cdc-0ad31c7566c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:05.212496 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2486259-8311-4ac7-8e24-4c3482bbe448 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:05.212761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeaf69c-81f1-4a36-9cdc-0ad31c7566c2 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:05.213206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeaf69c-81f1-4a36-9cdc-0ad31c7566c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:05.213693 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeaf69c-81f1-4a36-9cdc-0ad31c7566c2 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:05.214925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeaf69c-81f1-4a36-9cdc-0ad31c7566c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:05.215475 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeaf69c-81f1-4a36-9cdc-0ad31c7566c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:05.219660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeaf69c-81f1-4a36-9cdc-0ad31c7566c2 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:05.233233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-c2486259-8311-4ac7-8e24-4c3482bbe448 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 14:16:05.233233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:05.233233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:16:05.233233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:16:05.233233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:16:05.233233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:16:05.233233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:16:05.233233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:05.233233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 14:16:05.233233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:05.233233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:05.233233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:05.233233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:05.233233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:05.233233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:05.233233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:05.235044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:05.235044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:05.235044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:05.235044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:05.235044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:05.235044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:05.235044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:05.235044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:16:05.235044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:16:05.235044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:16:05.235044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:16:05.235044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:05.235044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:05.235044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:05.235044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:05.235044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:05.235044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:05.237475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:05.237475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:05.237475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:05.237475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:05.237475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:05.237475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:05.237475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:05.237475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:05.237475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:05.237475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:05.237475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:16:05.237475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:05.237475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 14:16:05.237475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:05.247314 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 435/865] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 348 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:05.247892 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 430/865] 10.222.0.64 () {56 vars in 1132 bytes} [Tue Oct 4 14:16:04 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:05.252747 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2486259-8311-4ac7-8e24-4c3482bbe448 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:05.253749 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeaf69c-81f1-4a36-9cdc-0ad31c7566c2 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:05.281010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:05.427910 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:05.484729 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2486259-8311-4ac7-8e24-4c3482bbe448 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:05.485649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeaf69c-81f1-4a36-9cdc-0ad31c7566c2 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:05.563302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ecd9886-11f5-4662-bbea-6159439c076c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:05.564424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ecd9886-11f5-4662-bbea-6159439c076c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:05.564912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ecd9886-11f5-4662-bbea-6159439c076c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:05.565343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ad5e06-197d-4a67-a98d-7a4a5a84231b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:05.565548 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ecd9886-11f5-4662-bbea-6159439c076c None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:05.566595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ecd9886-11f5-4662-bbea-6159439c076c None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:05.567107 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ecd9886-11f5-4662-bbea-6159439c076c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:05.567442 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ad5e06-197d-4a67-a98d-7a4a5a84231b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:05.568172 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ad5e06-197d-4a67-a98d-7a4a5a84231b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:05.569113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ad5e06-197d-4a67-a98d-7a4a5a84231b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:05.569703 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ecd9886-11f5-4662-bbea-6159439c076c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:05.592345 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 431/866] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Oct 4 14:16:05 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 334 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:05.594556 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ad5e06-197d-4a67-a98d-7a4a5a84231b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:05.597319 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ecd9886-11f5-4662-bbea-6159439c076c None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:05.597572 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ad5e06-197d-4a67-a98d-7a4a5a84231b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:05.622078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-23ad5e06-197d-4a67-a98d-7a4a5a84231b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:05.622078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:05.622078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:05.622078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:05.622078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:05.622078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:05.622078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:05.622078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:05.622078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:05.622078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:05.622078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:05.622078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:05.622078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:16:05.622078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:16:05.622078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:16:05.624365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:16:05.624365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:05.624365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:05.624365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:05.624365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:05.624365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:05.624365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:05.624365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:05.624365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:05.624365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:05.624365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:05.624365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:05.624365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:05.624365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:05.624365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:05.624365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:05.624365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:05.626028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:16:05.626028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:16:05.626028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:16:05.626028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:16:05.626028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:16:05.626028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:16:05.626028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:05.626028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:05.626028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 436/867] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:16:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 375 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:05.634932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ad5e06-197d-4a67-a98d-7a4a5a84231b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:05.768798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:05.827815 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ad5e06-197d-4a67-a98d-7a4a5a84231b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:05.832881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ecd9886-11f5-4662-bbea-6159439c076c None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:05.869498 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2b6acb-c6fd-4617-8228-36bec5fb0631 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:05.870239 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b6acb-c6fd-4617-8228-36bec5fb0631 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:05.870582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b6acb-c6fd-4617-8228-36bec5fb0631 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:05.870944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b6acb-c6fd-4617-8228-36bec5fb0631 None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:05.871727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2b6acb-c6fd-4617-8228-36bec5fb0631 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:05.872098 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2b6acb-c6fd-4617-8228-36bec5fb0631 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:05.873769 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2b6acb-c6fd-4617-8228-36bec5fb0631 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:05.892362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 437/868] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:16:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:05.898141 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2b6acb-c6fd-4617-8228-36bec5fb0631 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:05.911715 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4844a91c-4ee2-4da0-b9b6-ef0597a999f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:05.913392 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4844a91c-4ee2-4da0-b9b6-ef0597a999f5 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:05.914148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4844a91c-4ee2-4da0-b9b6-ef0597a999f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:05.914899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4844a91c-4ee2-4da0-b9b6-ef0597a999f5 None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:05.916346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4844a91c-4ee2-4da0-b9b6-ef0597a999f5 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:05.917535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4844a91c-4ee2-4da0-b9b6-ef0597a999f5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:05.921617 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4844a91c-4ee2-4da0-b9b6-ef0597a999f5 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:05.932033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 432/869] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Oct 4 14:16:05 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 336 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:06.025567 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2b6acb-c6fd-4617-8228-36bec5fb0631 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:06.065519 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:06.071768 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66a25b8-6908-45eb-929f-00d7f5d67a9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:06.075411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66a25b8-6908-45eb-929f-00d7f5d67a9f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:06.076740 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66a25b8-6908-45eb-929f-00d7f5d67a9f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:06.077398 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66a25b8-6908-45eb-929f-00d7f5d67a9f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:06.079058 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66a25b8-6908-45eb-929f-00d7f5d67a9f None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=5980e353164c4873bcdd2af847e0a1f1) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:06.079874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66a25b8-6908-45eb-929f-00d7f5d67a9f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:06.083030 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66a25b8-6908-45eb-929f-00d7f5d67a9f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:06.095565 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 438/870] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:16:05 2022] GET /identity/v3/role_assignments?user.id=5980e353164c4873bcdd2af847e0a1f1&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 608 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:06.462003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c7d34f-147b-4ed1-a847-b8949dbc5113 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:06.463340 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c7d34f-147b-4ed1-a847-b8949dbc5113 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:06.463833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c7d34f-147b-4ed1-a847-b8949dbc5113 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:06.466608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 433/871] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:06 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:06.473139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c80406-4a71-4a88-86e7-abee0513bf6e None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:06.473764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c80406-4a71-4a88-86e7-abee0513bf6e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:06.474231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c80406-4a71-4a88-86e7-abee0513bf6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:06.496873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-66c80406-4a71-4a88-86e7-abee0513bf6e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:06.499097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-66c80406-4a71-4a88-86e7-abee0513bf6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:06.580517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 439/872] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:06 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:16:06.871117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c7d34f-147b-4ed1-a847-b8949dbc5113 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:06.884142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:07.002496 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c7d34f-147b-4ed1-a847-b8949dbc5113 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:07.067564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74964490-36eb-41e5-b858-924297b358ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:07.068245 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74964490-36eb-41e5-b858-924297b358ff None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:07.068564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74964490-36eb-41e5-b858-924297b358ff None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:07.068837 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74964490-36eb-41e5-b858-924297b358ff None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:07.082189 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74964490-36eb-41e5-b858-924297b358ff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:16:07.087186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-74964490-36eb-41e5-b858-924297b358ff None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 14:16:07.087186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:07.087186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:07.087186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:07.087186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:07.087186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:07.087186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:07.087186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:07.087186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:07.087186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:07.087186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:07.087186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:07.087186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:16:07.087186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:16:07.087186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:16:07.087186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:16:07.088992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:16:07.088992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:16:07.088992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:16:07.088992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:16:07.088992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:07.088992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:07.088992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:07.088992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:07.088992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:07.088992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:07.088992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:07.088992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:07.088992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:07.088992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:07.088992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:07.088992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:07.090661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:07.090661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:07.090661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:07.090661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:07.090661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:16:07.090661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:16:07.090661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:16:07.090661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:16:07.090661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:16:07.090661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:16:07.090661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 14:16:07.090661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:07.090661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 434/873] 10.222.0.64 () {56 vars in 1120 bytes} [Tue Oct 4 14:16:06 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:07.095146 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c80406-4a71-4a88-86e7-abee0513bf6e None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:07.220074 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c80406-4a71-4a88-86e7-abee0513bf6e None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:07.252878 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:07.263597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e3e5da-89a5-4837-91b6-1dc2594bc7df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:07.264232 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e3e5da-89a5-4837-91b6-1dc2594bc7df None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:07.264672 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e3e5da-89a5-4837-91b6-1dc2594bc7df None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:07.264960 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e3e5da-89a5-4837-91b6-1dc2594bc7df None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:07.265651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e3e5da-89a5-4837-91b6-1dc2594bc7df None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:07.265959 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e3e5da-89a5-4837-91b6-1dc2594bc7df None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:07.267388 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e3e5da-89a5-4837-91b6-1dc2594bc7df None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:07.277908 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 440/874] 10.222.0.64 () {56 vars in 1115 bytes} [Tue Oct 4 14:16:07 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:07.285529 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74964490-36eb-41e5-b858-924297b358ff None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:07.423880 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:07.502857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74964490-36eb-41e5-b858-924297b358ff None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:07.561269 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b62d9d-7a7e-4f57-b9a4-5f0a68e24b54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:07.562369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b62d9d-7a7e-4f57-b9a4-5f0a68e24b54 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:07.562861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b62d9d-7a7e-4f57-b9a4-5f0a68e24b54 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:07.563340 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b62d9d-7a7e-4f57-b9a4-5f0a68e24b54 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:07.585554 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b62d9d-7a7e-4f57-b9a4-5f0a68e24b54 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:07.588357 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b62d9d-7a7e-4f57-b9a4-5f0a68e24b54 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:07.618271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-16b62d9d-7a7e-4f57-b9a4-5f0a68e24b54 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:07.618271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:07.618271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:07.618271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:07.618271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:07.618271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:07.618271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:07.618271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:07.618271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:07.618271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:07.618271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:07.618271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:07.618271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:07.618271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:07.618271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:07.618271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:07.619328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:07.619328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:07.619328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:07.619328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:07.619328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:07.619328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:07.619328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:07.619328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:07.619328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:07.619328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:07.619328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:07.619328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:07.619328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:07.619328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:07.619328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:07.619328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:07.619328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:07.620245 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:07.620245 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:07.620245 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:07.620878 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 435/875] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 336 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:07.629562 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e3e5da-89a5-4837-91b6-1dc2594bc7df None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:07.843113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e3e5da-89a5-4837-91b6-1dc2594bc7df None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:07.917663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1cd03e-bd6c-430b-83dc-3b4f42f80619 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:07.918836 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1cd03e-bd6c-430b-83dc-3b4f42f80619 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:07.919356 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1cd03e-bd6c-430b-83dc-3b4f42f80619 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:07.921359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1cd03e-bd6c-430b-83dc-3b4f42f80619 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:07.922664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1cd03e-bd6c-430b-83dc-3b4f42f80619 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:07.923213 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1cd03e-bd6c-430b-83dc-3b4f42f80619 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:07.925828 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1cd03e-bd6c-430b-83dc-3b4f42f80619 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:07.953432 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 441/876] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:07.962103 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b62d9d-7a7e-4f57-b9a4-5f0a68e24b54 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:08.035935 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fc859f-6368-4cec-b099-b77d7f13dc53 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:08.036465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fc859f-6368-4cec-b099-b77d7f13dc53 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:08.037202 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fc859f-6368-4cec-b099-b77d7f13dc53 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:08.040361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 442/877] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:08 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:08.049989 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51383bf9-961d-4964-96de-2811b0f31059 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:08.050495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51383bf9-961d-4964-96de-2811b0f31059 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:08.050987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51383bf9-961d-4964-96de-2811b0f31059 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:08.087188 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-51383bf9-961d-4964-96de-2811b0f31059 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:08.089459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-51383bf9-961d-4964-96de-2811b0f31059 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:08.159914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b62d9d-7a7e-4f57-b9a4-5f0a68e24b54 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:08.180692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:08.212295 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:08.244671 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 443/878] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:08 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:08.247131 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007b7563-852e-4d5a-ad75-3785a2501692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:08.249139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007b7563-852e-4d5a-ad75-3785a2501692 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:08.249867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007b7563-852e-4d5a-ad75-3785a2501692 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:08.250586 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007b7563-852e-4d5a-ad75-3785a2501692 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:08.264911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007b7563-852e-4d5a-ad75-3785a2501692 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:08.267851 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007b7563-852e-4d5a-ad75-3785a2501692 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:08.292251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-007b7563-852e-4d5a-ad75-3785a2501692 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 14:16:08.292251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:08.292251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:08.292251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:08.292251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:08.292251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:08.292251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:08.292251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:08.292251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:08.292251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:08.292251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:08.292251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:08.292251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:16:08.292251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:16:08.292251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:16:08.292251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:08.293909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:08.296139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:08.296139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:08.296139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:16:08.296139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:16:08.296139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:16:08.296139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:16:08.296139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:16:08.296139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:16:08.296139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 14:16:08.296139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:08.296139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 436/879] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Oct 4 14:16:07 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 327 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:08.303304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51383bf9-961d-4964-96de-2811b0f31059 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:08.490605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51383bf9-961d-4964-96de-2811b0f31059 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:08.559030 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df8faa5-c9c9-42d1-9903-25c501acae45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:08.560656 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df8faa5-c9c9-42d1-9903-25c501acae45 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:08.561152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df8faa5-c9c9-42d1-9903-25c501acae45 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:08.561621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df8faa5-c9c9-42d1-9903-25c501acae45 None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:08.562824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df8faa5-c9c9-42d1-9903-25c501acae45 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:08.563301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df8faa5-c9c9-42d1-9903-25c501acae45 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:08.565616 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df8faa5-c9c9-42d1-9903-25c501acae45 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:08.600839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 444/880] 10.222.0.64 () {56 vars in 1149 bytes} [Tue Oct 4 14:16:08 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:08.610524 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007b7563-852e-4d5a-ad75-3785a2501692 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:08.753455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df8faa5-c9c9-42d1-9903-25c501acae45 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:08.781112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007b7563-852e-4d5a-ad75-3785a2501692 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:08.823276 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:08.833544 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a206e4a-2d40-49c3-8e41-8b8e5ea30a9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:08.834189 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a206e4a-2d40-49c3-8e41-8b8e5ea30a9c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:08.834497 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a206e4a-2d40-49c3-8e41-8b8e5ea30a9c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:08.834774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a206e4a-2d40-49c3-8e41-8b8e5ea30a9c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:08.902960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:08.947621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df8faa5-c9c9-42d1-9903-25c501acae45 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:08.948692 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a206e4a-2d40-49c3-8e41-8b8e5ea30a9c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:08.951385 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a206e4a-2d40-49c3-8e41-8b8e5ea30a9c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:08.995861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4891e40-b96b-4a13-b209-9fea5a6695e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:08.996805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4891e40-b96b-4a13-b209-9fea5a6695e7 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:08.997167 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4891e40-b96b-4a13-b209-9fea5a6695e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:08.997625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4891e40-b96b-4a13-b209-9fea5a6695e7 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:08.998734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4891e40-b96b-4a13-b209-9fea5a6695e7 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:09.000618 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4891e40-b96b-4a13-b209-9fea5a6695e7 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:09.019377 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 445/881] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Oct 4 14:16:08 2022] POST /identity/v3/services => generated 251 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:16:09.021925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf09bc11-9cf2-44c0-8243-a2aaf14e491b None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:09.022296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf09bc11-9cf2-44c0-8243-a2aaf14e491b None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:09.022646 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf09bc11-9cf2-44c0-8243-a2aaf14e491b None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:09.024510 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 446/882] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:09.031699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13aa347-35fa-46f9-a088-1c86e8d24ba6 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:09.032227 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13aa347-35fa-46f9-a088-1c86e8d24ba6 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:09.032723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13aa347-35fa-46f9-a088-1c86e8d24ba6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:09.057982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-d13aa347-35fa-46f9-a088-1c86e8d24ba6 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:09.059833 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d13aa347-35fa-46f9-a088-1c86e8d24ba6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:09.117295 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 437/883] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:16:08 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 508 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 14:16:09.121805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92883bd4-e5c4-4fa4-8e5b-848201c0160e None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:09.122262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92883bd4-e5c4-4fa4-8e5b-848201c0160e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:09.122735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92883bd4-e5c4-4fa4-8e5b-848201c0160e None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:09.125607 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 438/884] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:09 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:09.127028 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92883bd4-e5c4-4fa4-8e5b-848201c0160e None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:09.182744 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 447/885] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:09 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:09.184369 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf77a335-0687-4ad8-890b-b8a859914752 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:09.184700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf77a335-0687-4ad8-890b-b8a859914752 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:09.185610 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf77a335-0687-4ad8-890b-b8a859914752 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:09.211846 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-bf77a335-0687-4ad8-890b-b8a859914752 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:09.213691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-bf77a335-0687-4ad8-890b-b8a859914752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:09.241032 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:09.300376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:09.335109 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92883bd4-e5c4-4fa4-8e5b-848201c0160e None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:09.344279 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 448/886] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:09 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:09.383910 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127bf609-daf3-43bd-9934-d903f47b84eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:09.383910 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127bf609-daf3-43bd-9934-d903f47b84eb None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:09.383910 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127bf609-daf3-43bd-9934-d903f47b84eb None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:09.383910 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127bf609-daf3-43bd-9934-d903f47b84eb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:09.411844 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127bf609-daf3-43bd-9934-d903f47b84eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:09.415186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127bf609-daf3-43bd-9934-d903f47b84eb None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:09.447263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-127bf609-daf3-43bd-9934-d903f47b84eb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:09.447263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:09.447263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:09.447263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:09.447263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:09.447263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:09.447263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:09.447263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:09.447263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:09.447263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:09.447263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:09.447263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:09.447263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:09.447263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:09.447263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:09.447263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:09.449162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:09.449162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:09.449162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:09.449162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:09.449162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:09.449162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:09.449162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:09.449162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:09.449162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:09.449162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:09.449162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:09.449162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:09.449162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:09.449162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:09.449162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:09.449162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:09.449162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:09.451277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:09.451277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:09.451277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:09.455138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 439/887] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 329 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:09.461173 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf77a335-0687-4ad8-890b-b8a859914752 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:09.487022 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127bf609-daf3-43bd-9934-d903f47b84eb None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:09.633416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127bf609-daf3-43bd-9934-d903f47b84eb None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:09.689006 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:09.698268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20d4d70-6d71-4333-8022-9b0fa460e87c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:09.699243 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20d4d70-6d71-4333-8022-9b0fa460e87c None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:09.699716 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20d4d70-6d71-4333-8022-9b0fa460e87c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:09.700179 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20d4d70-6d71-4333-8022-9b0fa460e87c None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:09.701488 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf77a335-0687-4ad8-890b-b8a859914752 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:09.701904 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20d4d70-6d71-4333-8022-9b0fa460e87c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:09.704387 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20d4d70-6d71-4333-8022-9b0fa460e87c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:09.733744 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 440/888] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Oct 4 14:16:09 2022] POST /identity/v3/services => generated 269 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:16:09.763438 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251fc804-3be4-4975-a655-5e6ac22cb568 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:09.764464 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251fc804-3be4-4975-a655-5e6ac22cb568 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:09.764902 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251fc804-3be4-4975-a655-5e6ac22cb568 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:09.765330 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251fc804-3be4-4975-a655-5e6ac22cb568 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:09.766245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251fc804-3be4-4975-a655-5e6ac22cb568 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:09.766721 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251fc804-3be4-4975-a655-5e6ac22cb568 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:09.768522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251fc804-3be4-4975-a655-5e6ac22cb568 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:09.783517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 449/889] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 323 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:09.789028 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20d4d70-6d71-4333-8022-9b0fa460e87c None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:09.799694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251fc804-3be4-4975-a655-5e6ac22cb568 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:09.823313 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:09.987575 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20d4d70-6d71-4333-8022-9b0fa460e87c None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:09.998771 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251fc804-3be4-4975-a655-5e6ac22cb568 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:10.056841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a48cb45-365f-4b57-9e99-d5bfe12da5f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:10.058338 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a48cb45-365f-4b57-9e99-d5bfe12da5f9 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:10.059062 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a48cb45-365f-4b57-9e99-d5bfe12da5f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:10.059254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad85f8d-efb5-4aa6-a649-aad6b8a5578c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:10.060055 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a48cb45-365f-4b57-9e99-d5bfe12da5f9 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:10.060566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad85f8d-efb5-4aa6-a649-aad6b8a5578c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:10.061415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad85f8d-efb5-4aa6-a649-aad6b8a5578c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:10.061887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad85f8d-efb5-4aa6-a649-aad6b8a5578c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:10.062565 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a48cb45-365f-4b57-9e99-d5bfe12da5f9 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:10.065698 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a48cb45-365f-4b57-9e99-d5bfe12da5f9 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:10.076032 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad85f8d-efb5-4aa6-a649-aad6b8a5578c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:10.078772 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad85f8d-efb5-4aa6-a649-aad6b8a5578c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:10.084855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-5a48cb45-365f-4b57-9e99-d5bfe12da5f9 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 14:16:10.084855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:10.084855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:16:10.084855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:16:10.084855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:16:10.084855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:16:10.084855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:16:10.084855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:10.084855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 14:16:10.084855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:10.084855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:10.084855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:10.084855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:10.084855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:10.084855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:10.084855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:10.084855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:10.086894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:10.086894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:10.086894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:10.086894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:10.086894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:10.086894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:10.086894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:16:10.086894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:16:10.086894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:16:10.086894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:16:10.086894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:10.086894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:10.086894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:10.086894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:10.086894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:10.086894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:10.089421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:10.089421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:10.089421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:10.089421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:10.089421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:10.089421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:10.089421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:10.089421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:10.089421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:10.089421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:10.089421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:16:10.089421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:10.089421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 14:16:10.089421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:10.094415 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 450/890] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Oct 4 14:16:09 2022] GET /identity/v3/services/object-store => generated 93 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:10.104183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a48cb45-365f-4b57-9e99-d5bfe12da5f9 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:10.105282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-bad85f8d-efb5-4aa6-a649-aad6b8a5578c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:10.105282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:10.105282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:10.105282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:10.105282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:10.105282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:10.105282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:10.105282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:10.105282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:10.105282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:10.105282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:10.105282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:10.105282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:16:10.105282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:16:10.105282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:16:10.106993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:16:10.106993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:10.106993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:10.106993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:10.106993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:10.106993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:10.106993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:10.106993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:10.106993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:10.106993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:10.106993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:10.106993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:10.106993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:10.106993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:10.106993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:10.106993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:10.106993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:10.108821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:16:10.108821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:16:10.108821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:16:10.108821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:16:10.108821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:16:10.108821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:16:10.108821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:10.108821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:10.113679 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 441/891] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:16:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 325 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:10.124175 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad85f8d-efb5-4aa6-a649-aad6b8a5578c None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:10.186440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:10.248127 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:10.249715 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad85f8d-efb5-4aa6-a649-aad6b8a5578c None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:10.295379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4830df46-cf2c-4ef4-a211-e93fd0a186ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:10.296460 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4830df46-cf2c-4ef4-a211-e93fd0a186ca None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:10.296889 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4830df46-cf2c-4ef4-a211-e93fd0a186ca None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:10.297311 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4830df46-cf2c-4ef4-a211-e93fd0a186ca None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:10.298166 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4830df46-cf2c-4ef4-a211-e93fd0a186ca None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:10.298620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4830df46-cf2c-4ef4-a211-e93fd0a186ca None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:10.300711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4830df46-cf2c-4ef4-a211-e93fd0a186ca None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:10.313756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a48cb45-365f-4b57-9e99-d5bfe12da5f9 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:10.321113 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 442/892] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:16:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:10.322808 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4830df46-cf2c-4ef4-a211-e93fd0a186ca None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:10.392158 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87cd10e-8c79-42df-be41-1506aaa4c135 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:10.393276 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87cd10e-8c79-42df-be41-1506aaa4c135 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:10.393764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87cd10e-8c79-42df-be41-1506aaa4c135 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:10.394252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87cd10e-8c79-42df-be41-1506aaa4c135 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:10.395530 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87cd10e-8c79-42df-be41-1506aaa4c135 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:10.396071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87cd10e-8c79-42df-be41-1506aaa4c135 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:10.399131 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87cd10e-8c79-42df-be41-1506aaa4c135 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:10.415790 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 451/893] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Oct 4 14:16:10 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:10.420400 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87cd10e-8c79-42df-be41-1506aaa4c135 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:10.446867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4830df46-cf2c-4ef4-a211-e93fd0a186ca None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:10.464328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:10.491123 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3f40de-4b1a-4fac-b192-8dc825a5e727 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:10.492022 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3f40de-4b1a-4fac-b192-8dc825a5e727 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:10.492483 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3f40de-4b1a-4fac-b192-8dc825a5e727 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:10.492897 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3f40de-4b1a-4fac-b192-8dc825a5e727 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:10.493789 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3f40de-4b1a-4fac-b192-8dc825a5e727 None admin] RBAC: Adding query filter params (role.id=47e35bb0c3b44184986f4e28032e8c17, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=34afa4aac10c4a018ca05a0ba34c940a) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:10.494428 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3f40de-4b1a-4fac-b192-8dc825a5e727 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:10.496372 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3f40de-4b1a-4fac-b192-8dc825a5e727 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:10.505415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 443/894] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:16:10 2022] GET /identity/v3/role_assignments?user.id=34afa4aac10c4a018ca05a0ba34c940a&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=47e35bb0c3b44184986f4e28032e8c17 => generated 259 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:10.577305 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87cd10e-8c79-42df-be41-1506aaa4c135 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:10.641932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537f65e0-e479-491b-91f7-d9bf82f31880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:10.643068 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537f65e0-e479-491b-91f7-d9bf82f31880 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:10.643566 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537f65e0-e479-491b-91f7-d9bf82f31880 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:10.644049 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537f65e0-e479-491b-91f7-d9bf82f31880 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:10.645428 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537f65e0-e479-491b-91f7-d9bf82f31880 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:10.646044 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537f65e0-e479-491b-91f7-d9bf82f31880 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:10.648868 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537f65e0-e479-491b-91f7-d9bf82f31880 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:10.651552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:10.665053 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 452/895] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Oct 4 14:16:10 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:11.536895 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ce8d8d-d378-405a-bc21-3ae98b9c4f34 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:11.537754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ce8d8d-d378-405a-bc21-3ae98b9c4f34 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:11.537754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ce8d8d-d378-405a-bc21-3ae98b9c4f34 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:11.539199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 444/896] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:11.545906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af098ff4-6946-4095-b1a9-5cb43cdb8264 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:11.546674 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af098ff4-6946-4095-b1a9-5cb43cdb8264 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:11.546674 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af098ff4-6946-4095-b1a9-5cb43cdb8264 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:11.566649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-af098ff4-6946-4095-b1a9-5cb43cdb8264 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:11.567947 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-af098ff4-6946-4095-b1a9-5cb43cdb8264 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:11.660911 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 453/897] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:11 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:16:11.947902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ce8d8d-d378-405a-bc21-3ae98b9c4f34 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:12.078228 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ce8d8d-d378-405a-bc21-3ae98b9c4f34 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:12.121059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49859db-7003-4252-b118-2c4c12de05ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:12.121717 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49859db-7003-4252-b118-2c4c12de05ed None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:12.122001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49859db-7003-4252-b118-2c4c12de05ed None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:12.122734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49859db-7003-4252-b118-2c4c12de05ed None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:12.123372 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49859db-7003-4252-b118-2c4c12de05ed None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:12.124838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49859db-7003-4252-b118-2c4c12de05ed None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:12.135251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-a49859db-7003-4252-b118-2c4c12de05ed None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 14:16:12.135251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:12.135251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:16:12.135251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:16:12.135251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:16:12.135251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:16:12.135251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:16:12.135251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:12.135251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 14:16:12.135251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:12.135251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:12.135251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:12.135251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:12.135251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:12.135251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:12.135251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:12.135251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:12.137231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:12.137231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:12.137231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:12.137231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:12.137231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:12.137231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:12.137231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:16:12.137231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:16:12.137231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:16:12.137231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:16:12.137231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:12.137231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:12.137231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:12.137231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:12.137231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:12.137231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:12.139576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:12.139576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:12.139576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:12.139576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:12.139576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:12.139576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:12.139576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:12.139576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:12.139576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:12.139576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:12.139576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:16:12.139576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:12.139576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 14:16:12.139576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:12.139576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 445/898] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:16:11 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:12.145826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af098ff4-6946-4095-b1a9-5cb43cdb8264 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:12.265520 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af098ff4-6946-4095-b1a9-5cb43cdb8264 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:12.307229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:12.314630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6798713f-f9cd-4786-9f68-4077110b4c95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:12.315264 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6798713f-f9cd-4786-9f68-4077110b4c95 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:12.315616 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6798713f-f9cd-4786-9f68-4077110b4c95 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:12.315894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6798713f-f9cd-4786-9f68-4077110b4c95 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:12.316578 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6798713f-f9cd-4786-9f68-4077110b4c95 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:12.316886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6798713f-f9cd-4786-9f68-4077110b4c95 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:12.318310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6798713f-f9cd-4786-9f68-4077110b4c95 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:12.324684 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 454/899] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:16:12 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:12.337755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49859db-7003-4252-b118-2c4c12de05ed None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:12.384256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:12.465308 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49859db-7003-4252-b118-2c4c12de05ed None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:12.501229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5901f2-34f7-427f-9f12-73df7fabd789 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:12.501866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5901f2-34f7-427f-9f12-73df7fabd789 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:12.502151 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5901f2-34f7-427f-9f12-73df7fabd789 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:12.502427 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5901f2-34f7-427f-9f12-73df7fabd789 None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:12.503074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5901f2-34f7-427f-9f12-73df7fabd789 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:12.503379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5901f2-34f7-427f-9f12-73df7fabd789 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:12.504898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5901f2-34f7-427f-9f12-73df7fabd789 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:12.511169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 446/900] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:16:12 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:12.517084 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6798713f-f9cd-4786-9f68-4077110b4c95 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:12.643446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6798713f-f9cd-4786-9f68-4077110b4c95 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:12.687238 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895483d4-ef06-4acf-a9bd-49bcf605f040 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:12.687862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895483d4-ef06-4acf-a9bd-49bcf605f040 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:12.688135 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895483d4-ef06-4acf-a9bd-49bcf605f040 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:12.688433 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895483d4-ef06-4acf-a9bd-49bcf605f040 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:12.689136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895483d4-ef06-4acf-a9bd-49bcf605f040 None admin] RBAC: Adding query filter params (interface=public, service_id=60bc888dc2504030852ce376a1540a94, region_id=RegionOne) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:12.689445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895483d4-ef06-4acf-a9bd-49bcf605f040 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:12.690862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895483d4-ef06-4acf-a9bd-49bcf605f040 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:12.697324 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 455/901] 10.222.0.64 () {56 vars in 1251 bytes} [Tue Oct 4 14:16:12 2022] GET /identity/v3/endpoints?service_id=60bc888dc2504030852ce376a1540a94&interface=public®ion_id=RegionOne => generated 194 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:12.703403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5901f2-34f7-427f-9f12-73df7fabd789 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:12.835243 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5901f2-34f7-427f-9f12-73df7fabd789 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:12.838191 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:12.880902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315730ea-af42-41a3-9276-ee0aa6227076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:12.881539 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315730ea-af42-41a3-9276-ee0aa6227076 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:12.881823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315730ea-af42-41a3-9276-ee0aa6227076 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:12.889659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315730ea-af42-41a3-9276-ee0aa6227076 None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:12.890405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315730ea-af42-41a3-9276-ee0aa6227076 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:12.891848 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315730ea-af42-41a3-9276-ee0aa6227076 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:12.899024 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 447/902] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Oct 4 14:16:12 2022] GET /identity/v3/services => generated 1268 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:16:12.910335 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b90e71-186d-4bc3-8b71-63e6ba722e30 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:12.910615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b90e71-186d-4bc3-8b71-63e6ba722e30 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:12.910920 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b90e71-186d-4bc3-8b71-63e6ba722e30 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:12.912357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 456/903] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:16:12.918737 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51ec864-15d8-4987-b52a-7fd84c5c800e None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:12.919018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51ec864-15d8-4987-b52a-7fd84c5c800e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:12.919849 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51ec864-15d8-4987-b52a-7fd84c5c800e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:12.941505 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-c51ec864-15d8-4987-b52a-7fd84c5c800e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:12.942808 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c51ec864-15d8-4987-b52a-7fd84c5c800e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:13.032405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 448/904] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:12 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:13.429915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b90e71-186d-4bc3-8b71-63e6ba722e30 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:13.441221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a26b579-db70-4b6f-a4ec-e4e33e3610e9 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:13.441736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a26b579-db70-4b6f-a4ec-e4e33e3610e9 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:13.442241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a26b579-db70-4b6f-a4ec-e4e33e3610e9 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:13.447075 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 449/905] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:13 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:13.458294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4651b73-3347-4304-8db9-8b0ecd6cbf15 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:13.458762 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4651b73-3347-4304-8db9-8b0ecd6cbf15 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:13.459847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4651b73-3347-4304-8db9-8b0ecd6cbf15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:13.501638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-a4651b73-3347-4304-8db9-8b0ecd6cbf15 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:13.504827 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a4651b73-3347-4304-8db9-8b0ecd6cbf15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:13.517539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:13.590800 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b90e71-186d-4bc3-8b71-63e6ba722e30 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:13.630070 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 450/906] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:13 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:13.658215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0fea18-02be-4470-8900-c2ac92606ec6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:13.659534 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0fea18-02be-4470-8900-c2ac92606ec6 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:13.660015 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0fea18-02be-4470-8900-c2ac92606ec6 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:13.660487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0fea18-02be-4470-8900-c2ac92606ec6 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:13.661304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0fea18-02be-4470-8900-c2ac92606ec6 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:13.662836 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0fea18-02be-4470-8900-c2ac92606ec6 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:13.679001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-1e0fea18-02be-4470-8900-c2ac92606ec6 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 14:16:13.679001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:13.679001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:16:13.679001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:16:13.679001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:16:13.679001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:16:13.679001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:16:13.679001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:13.679001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 14:16:13.679001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:13.679001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:13.679001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:13.679001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:13.679001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:13.679001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:13.679001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:13.680278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:13.680278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:13.680278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:13.680278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:13.680278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:13.680278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:13.680278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:13.680278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:16:13.680278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:16:13.680278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:16:13.680278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:16:13.680278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:13.680278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:13.680278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:13.680278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:13.680278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:13.680278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:13.681372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:13.681372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:13.681372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:13.681372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:13.681372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:13.681372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:13.681372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:13.681372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:13.681372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:13.681372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:13.681372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:16:13.681372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:13.681372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 14:16:13.681372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:13.688491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 457/907] 10.222.0.64 () {56 vars in 1132 bytes} [Tue Oct 4 14:16:13 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:13.692655 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4651b73-3347-4304-8db9-8b0ecd6cbf15 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:13.748915 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:13.890087 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4651b73-3347-4304-8db9-8b0ecd6cbf15 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:13.957789 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2198a62e-c1b1-4b99-9e75-285b747508f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:13.958566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2198a62e-c1b1-4b99-9e75-285b747508f0 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:13.958973 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2198a62e-c1b1-4b99-9e75-285b747508f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:13.959379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2198a62e-c1b1-4b99-9e75-285b747508f0 None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:13.960337 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2198a62e-c1b1-4b99-9e75-285b747508f0 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:13.960770 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2198a62e-c1b1-4b99-9e75-285b747508f0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:13.962908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2198a62e-c1b1-4b99-9e75-285b747508f0 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:13.976116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 451/908] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Oct 4 14:16:13 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:13.982259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0fea18-02be-4470-8900-c2ac92606ec6 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:14.175313 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0fea18-02be-4470-8900-c2ac92606ec6 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:14.178936 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2198a62e-c1b1-4b99-9e75-285b747508f0 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:14.222714 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:14.255100 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd01bd1-ec0c-4c5b-97fe-d78745902573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:14.255756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd01bd1-ec0c-4c5b-97fe-d78745902573 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:14.256028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd01bd1-ec0c-4c5b-97fe-d78745902573 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:14.256292 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd01bd1-ec0c-4c5b-97fe-d78745902573 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:14.257019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd01bd1-ec0c-4c5b-97fe-d78745902573 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:14.257379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd01bd1-ec0c-4c5b-97fe-d78745902573 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:14.258821 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd01bd1-ec0c-4c5b-97fe-d78745902573 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:14.266130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 458/909] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Oct 4 14:16:13 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:14.278185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd01bd1-ec0c-4c5b-97fe-d78745902573 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:14.376506 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:14.391199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2198a62e-c1b1-4b99-9e75-285b747508f0 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:14.417971 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd01bd1-ec0c-4c5b-97fe-d78745902573 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:14.456419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f62ebe0-4378-405a-aa49-38648e91d816 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:14.457476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f62ebe0-4378-405a-aa49-38648e91d816 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:14.457948 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f62ebe0-4378-405a-aa49-38648e91d816 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:14.458436 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f62ebe0-4378-405a-aa49-38648e91d816 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:14.470297 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b284d84d-ea75-4131-afe8-a17d394dae74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:14.471295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b284d84d-ea75-4131-afe8-a17d394dae74 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:14.471599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b284d84d-ea75-4131-afe8-a17d394dae74 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:14.472014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b284d84d-ea75-4131-afe8-a17d394dae74 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:14.473183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b284d84d-ea75-4131-afe8-a17d394dae74 None admin] RBAC: Adding query filter params (interface=public, service_id=a6d67dbef4bc47a892297a74d7d61155, region_id=RegionOne) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:14.473715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b284d84d-ea75-4131-afe8-a17d394dae74 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:14.475906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b284d84d-ea75-4131-afe8-a17d394dae74 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:14.479049 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f62ebe0-4378-405a-aa49-38648e91d816 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:16:14.485050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 459/910] 10.222.0.64 () {56 vars in 1251 bytes} [Tue Oct 4 14:16:14 2022] GET /identity/v3/endpoints?service_id=a6d67dbef4bc47a892297a74d7d61155&interface=public®ion_id=RegionOne => generated 194 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:14.486117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-5f62ebe0-4378-405a-aa49-38648e91d816 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 14:16:14.486117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:14.486117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:14.486117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:14.486117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:14.486117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:14.486117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:14.486117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:14.486117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:14.486117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:14.486117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:14.486117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:14.486117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:16:14.486117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:16:14.486117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:16:14.486117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:16:14.487266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:16:14.487266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:16:14.487266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:16:14.487266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:16:14.487266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:14.487266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:14.487266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:14.487266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:14.487266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:14.487266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:14.487266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:14.487266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:14.487266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:14.487266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:14.487266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:14.487266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:14.488159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:14.488159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:14.488159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:14.488159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:14.488159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:16:14.488159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:16:14.488159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:16:14.488159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:16:14.488159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:16:14.488159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:16:14.488159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 14:16:14.488159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:14.488847 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc39a72-e009-400b-983b-6c40c476ffd9 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:14.488847 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc39a72-e009-400b-983b-6c40c476ffd9 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:14.488847 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc39a72-e009-400b-983b-6c40c476ffd9 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:14.491189 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 452/911] 10.222.0.64 () {56 vars in 1120 bytes} [Tue Oct 4 14:16:14 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:14.496297 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 460/912] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:14 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:14.497896 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e6a335-d9a0-4f95-91c3-0aecb8c25810 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:14.498190 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e6a335-d9a0-4f95-91c3-0aecb8c25810 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:14.498453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e6a335-d9a0-4f95-91c3-0aecb8c25810 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:14.501325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc39a72-e009-400b-983b-6c40c476ffd9 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:14.519675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-a0e6a335-d9a0-4f95-91c3-0aecb8c25810 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:14.521110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a0e6a335-d9a0-4f95-91c3-0aecb8c25810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:14.600790 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:14.632012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 453/913] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:14 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:16:14.633200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e6a335-d9a0-4f95-91c3-0aecb8c25810 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:14.708826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc39a72-e009-400b-983b-6c40c476ffd9 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:14.779186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d146356-c537-46f9-9302-e9ff48c6cf4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:14.780014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d146356-c537-46f9-9302-e9ff48c6cf4a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:14.780288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d146356-c537-46f9-9302-e9ff48c6cf4a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:14.780676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d146356-c537-46f9-9302-e9ff48c6cf4a None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:14.781367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d146356-c537-46f9-9302-e9ff48c6cf4a None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:14.781671 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d146356-c537-46f9-9302-e9ff48c6cf4a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:14.783228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d146356-c537-46f9-9302-e9ff48c6cf4a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:14.797222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 461/914] 10.222.0.64 () {56 vars in 1115 bytes} [Tue Oct 4 14:16:14 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:14.805952 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d146356-c537-46f9-9302-e9ff48c6cf4a None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:14.814009 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:14.852944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e6a335-d9a0-4f95-91c3-0aecb8c25810 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:14.924019 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb0192a-97bb-48f7-8251-7ade41dd5c3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:14.925217 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb0192a-97bb-48f7-8251-7ade41dd5c3c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:14.925568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb0192a-97bb-48f7-8251-7ade41dd5c3c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:14.926054 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb0192a-97bb-48f7-8251-7ade41dd5c3c None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:14.927529 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb0192a-97bb-48f7-8251-7ade41dd5c3c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:14.930689 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb0192a-97bb-48f7-8251-7ade41dd5c3c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:14.944550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 454/915] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Oct 4 14:16:14 2022] GET /identity/v3/services => generated 1268 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:16:14.945190 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb0192a-97bb-48f7-8251-7ade41dd5c3c None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:14.947019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d146356-c537-46f9-9302-e9ff48c6cf4a None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:14.962379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:15.005439 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2bc0f5-3371-460a-ae4a-d366f8f873de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:15.006757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2bc0f5-3371-460a-ae4a-d366f8f873de None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:15.007682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2bc0f5-3371-460a-ae4a-d366f8f873de None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:15.009044 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2bc0f5-3371-460a-ae4a-d366f8f873de None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:15.027547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2bc0f5-3371-460a-ae4a-d366f8f873de None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:15.029903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2bc0f5-3371-460a-ae4a-d366f8f873de None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:15.061361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-1b2bc0f5-3371-460a-ae4a-d366f8f873de None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:15.061361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:15.061361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:15.061361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:15.061361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:15.061361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:15.061361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:15.061361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:15.061361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:15.061361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:15.061361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:15.061361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:15.061361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:15.061361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:15.061361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:15.061361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:15.063373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:15.063373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:15.063373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:15.063373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:15.063373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:15.063373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:15.063373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:15.063373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:15.063373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:15.063373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:15.063373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:15.063373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:15.063373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:15.063373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:15.063373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:15.063373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:15.063373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:15.065243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:15.065243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:15.065243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:15.065243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 462/916] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:15.073104 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2bc0f5-3371-460a-ae4a-d366f8f873de None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:15.140191 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb0192a-97bb-48f7-8251-7ade41dd5c3c None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:15.185150 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bdb961-2ee6-4792-9b27-919472735643 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:15.185907 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bdb961-2ee6-4792-9b27-919472735643 None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:15.186195 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bdb961-2ee6-4792-9b27-919472735643 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:15.186475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bdb961-2ee6-4792-9b27-919472735643 None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:15.187173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bdb961-2ee6-4792-9b27-919472735643 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:15.188712 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bdb961-2ee6-4792-9b27-919472735643 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:15.208221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 455/917] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Oct 4 14:16:14 2022] POST /identity/v3/services => generated 241 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:16:15.290769 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2bc0f5-3371-460a-ae4a-d366f8f873de None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:15.353501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8010648e-13bf-4cb6-aadd-ff23cb0e9fe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:15.354154 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8010648e-13bf-4cb6-aadd-ff23cb0e9fe0 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:15.354442 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8010648e-13bf-4cb6-aadd-ff23cb0e9fe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:15.354725 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8010648e-13bf-4cb6-aadd-ff23cb0e9fe0 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:15.355394 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8010648e-13bf-4cb6-aadd-ff23cb0e9fe0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:15.355701 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8010648e-13bf-4cb6-aadd-ff23cb0e9fe0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:15.357309 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8010648e-13bf-4cb6-aadd-ff23cb0e9fe0 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:15.373522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 463/918] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:15.391684 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bdb961-2ee6-4792-9b27-919472735643 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:15.402267 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:15.427262 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926938df-c5a4-4b9f-b2ab-d8cd9824e6b7 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:15.427698 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926938df-c5a4-4b9f-b2ab-d8cd9824e6b7 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:15.428895 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926938df-c5a4-4b9f-b2ab-d8cd9824e6b7 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:15.432369 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 464/919] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:15 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:15.439543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c029a6ba-72c6-47ef-a5c9-e07f74beb097 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:15.440027 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c029a6ba-72c6-47ef-a5c9-e07f74beb097 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:15.440915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c029a6ba-72c6-47ef-a5c9-e07f74beb097 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:15.483195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-c029a6ba-72c6-47ef-a5c9-e07f74beb097 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:15.487208 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c029a6ba-72c6-47ef-a5c9-e07f74beb097 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:15.507436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:15.585166 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bdb961-2ee6-4792-9b27-919472735643 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:15.585777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 465/920] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:15 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:15.649439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05bb0463-4b8e-412b-aa4f-ac73a18730ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:15.650156 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bb0463-4b8e-412b-aa4f-ac73a18730ad None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:15.650444 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bb0463-4b8e-412b-aa4f-ac73a18730ad None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:15.650716 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bb0463-4b8e-412b-aa4f-ac73a18730ad None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:15.660098 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05bb0463-4b8e-412b-aa4f-ac73a18730ad None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:15.661612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05bb0463-4b8e-412b-aa4f-ac73a18730ad None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:15.675395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-05bb0463-4b8e-412b-aa4f-ac73a18730ad None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 14:16:15.675395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:15.675395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:15.675395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:15.675395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:15.675395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:15.675395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:15.675395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:15.675395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:15.675395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:15.675395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:15.675395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:15.675395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:16:15.675395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:16:15.675395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:16:15.675395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:15.677284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:15.680671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:15.680671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:15.680671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:16:15.680671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:16:15.680671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:16:15.680671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:16:15.680671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:16:15.680671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:16:15.680671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 14:16:15.680671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:15.680671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 456/921] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Oct 4 14:16:15 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:15.693214 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c029a6ba-72c6-47ef-a5c9-e07f74beb097 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:15.877254 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c029a6ba-72c6-47ef-a5c9-e07f74beb097 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:15.917353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a94e34a-c44f-488d-9193-56fab2c1787d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:15.917986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a94e34a-c44f-488d-9193-56fab2c1787d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:15.918267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a94e34a-c44f-488d-9193-56fab2c1787d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:15.918541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a94e34a-c44f-488d-9193-56fab2c1787d None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:15.919887 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a94e34a-c44f-488d-9193-56fab2c1787d None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:15.920187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a94e34a-c44f-488d-9193-56fab2c1787d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:15.921696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a94e34a-c44f-488d-9193-56fab2c1787d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:15.942105 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 466/922] 10.222.0.64 () {56 vars in 1149 bytes} [Tue Oct 4 14:16:15 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:15.952605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05bb0463-4b8e-412b-aa4f-ac73a18730ad None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:16.034507 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:16.065825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a94e34a-c44f-488d-9193-56fab2c1787d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:16.076601 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:16.083884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05bb0463-4b8e-412b-aa4f-ac73a18730ad None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:16.122405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b0d61e-f8e5-4c90-a6ba-cfd4f1182899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:16.123058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b0d61e-f8e5-4c90-a6ba-cfd4f1182899 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:16.123339 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b0d61e-f8e5-4c90-a6ba-cfd4f1182899 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:16.123609 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b0d61e-f8e5-4c90-a6ba-cfd4f1182899 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:16.137399 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b0d61e-f8e5-4c90-a6ba-cfd4f1182899 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:16.138833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b0d61e-f8e5-4c90-a6ba-cfd4f1182899 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:16.156876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-c6b0d61e-f8e5-4c90-a6ba-cfd4f1182899 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:16.156876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:16.156876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:16.156876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:16.156876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:16.156876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:16.156876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:16.156876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:16.156876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:16.156876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:16.156876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:16.156876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:16.156876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:16.156876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:16.156876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:16.156876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:16.158755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:16.158755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:16.158755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:16.158755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:16.158755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:16.158755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:16.158755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:16.158755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:16.158755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:16.158755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:16.158755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:16.158755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:16.158755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:16.158755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:16.158755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:16.158755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:16.158755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:16.161375 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:16.161375 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:16.161375 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:16.161375 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 457/923] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:16.171428 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b0d61e-f8e5-4c90-a6ba-cfd4f1182899 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:16.192918 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a94e34a-c44f-488d-9193-56fab2c1787d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:16.252680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b156b17-c5e0-43bc-97e7-5f39bc878e5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:16.256027 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b156b17-c5e0-43bc-97e7-5f39bc878e5b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:16.257043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b156b17-c5e0-43bc-97e7-5f39bc878e5b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:16.257854 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b156b17-c5e0-43bc-97e7-5f39bc878e5b None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:16.259146 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b156b17-c5e0-43bc-97e7-5f39bc878e5b None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:16.263511 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b156b17-c5e0-43bc-97e7-5f39bc878e5b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:16.292799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-1b156b17-c5e0-43bc-97e7-5f39bc878e5b None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 14:16:16.292799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:16.292799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:16:16.292799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:16:16.292799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:16:16.292799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:16:16.292799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:16:16.292799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:16.292799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 14:16:16.292799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:16.292799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:16.292799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:16.292799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:16.292799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:16.292799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:16.292799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:16.292799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:16.294752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:16.294752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:16.294752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:16.294752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:16.294752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:16.294752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:16.294752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:16:16.294752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:16:16.294752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:16:16.294752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:16:16.294752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:16.294752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:16.294752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:16.294752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:16.294752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:16.294752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:16.296385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:16.296385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:16.296385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:16.296385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:16.296385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:16.296385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:16.296385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:16.296385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:16.296385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:16.296385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:16.296385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:16:16.296385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:16.296385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 14:16:16.296385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:16.299901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 467/924] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:16:16 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:16.306398 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b156b17-c5e0-43bc-97e7-5f39bc878e5b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:16.384278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b0d61e-f8e5-4c90-a6ba-cfd4f1182899 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:16.419941 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:16.443031 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:16.463627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d420fa-c404-4ff2-8ee0-c637235cb8b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:16.465203 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d420fa-c404-4ff2-8ee0-c637235cb8b0 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:16.466605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d420fa-c404-4ff2-8ee0-c637235cb8b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:16.467459 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d420fa-c404-4ff2-8ee0-c637235cb8b0 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:16.469126 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d420fa-c404-4ff2-8ee0-c637235cb8b0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:16.469930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d420fa-c404-4ff2-8ee0-c637235cb8b0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:16.472792 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d420fa-c404-4ff2-8ee0-c637235cb8b0 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:16.504266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 458/925] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:16.505379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b156b17-c5e0-43bc-97e7-5f39bc878e5b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:16.506523 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d420fa-c404-4ff2-8ee0-c637235cb8b0 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:16.572144 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd1101a-4f7a-41a7-81f3-8388a647ef26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:16.574243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd1101a-4f7a-41a7-81f3-8388a647ef26 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:16.574243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd1101a-4f7a-41a7-81f3-8388a647ef26 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:16.574243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd1101a-4f7a-41a7-81f3-8388a647ef26 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:16.576331 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd1101a-4f7a-41a7-81f3-8388a647ef26 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:16.576882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd1101a-4f7a-41a7-81f3-8388a647ef26 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:16.579439 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd1101a-4f7a-41a7-81f3-8388a647ef26 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:16.596472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 468/926] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:16:16 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:16.598443 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd1101a-4f7a-41a7-81f3-8388a647ef26 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:16.707308 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d420fa-c404-4ff2-8ee0-c637235cb8b0 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:16.773494 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b07aaef-204a-4dde-b937-ac5c8ebd6c55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:16.775029 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b07aaef-204a-4dde-b937-ac5c8ebd6c55 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:16.775814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b07aaef-204a-4dde-b937-ac5c8ebd6c55 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:16.776723 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b07aaef-204a-4dde-b937-ac5c8ebd6c55 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:16.790572 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b07aaef-204a-4dde-b937-ac5c8ebd6c55 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:16.793766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b07aaef-204a-4dde-b937-ac5c8ebd6c55 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:16.797742 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd1101a-4f7a-41a7-81f3-8388a647ef26 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:16.816503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-0b07aaef-204a-4dde-b937-ac5c8ebd6c55 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:16.816503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:16.816503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:16.816503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:16.816503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:16.816503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:16.816503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:16.816503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:16.816503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:16.816503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:16.816503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:16.816503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:16.816503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:16:16.816503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:16:16.816503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:16:16.818688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:16:16.818688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:16.818688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:16.818688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:16.818688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:16.818688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:16.818688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:16.818688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:16.818688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:16.818688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:16.818688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:16.818688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:16.818688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:16.818688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:16.818688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:16.818688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:16.818688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:16.820673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:16:16.820673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:16:16.820673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:16:16.820673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:16:16.820673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:16:16.820673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:16:16.820673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:16.820673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:16.831190 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 459/927] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:16:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:16.835721 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b07aaef-204a-4dde-b937-ac5c8ebd6c55 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:16.870112 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c805b357-192d-48a0-b4ff-8d98ad93515c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:16.870112 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c805b357-192d-48a0-b4ff-8d98ad93515c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:16.870112 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c805b357-192d-48a0-b4ff-8d98ad93515c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:16.870603 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c805b357-192d-48a0-b4ff-8d98ad93515c None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:16.871478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c805b357-192d-48a0-b4ff-8d98ad93515c None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:16.872540 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c805b357-192d-48a0-b4ff-8d98ad93515c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:16.875311 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c805b357-192d-48a0-b4ff-8d98ad93515c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:16.880329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:16.896869 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 469/928] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Oct 4 14:16:16 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:16.900514 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c805b357-192d-48a0-b4ff-8d98ad93515c None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:16.926299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:17.061423 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b07aaef-204a-4dde-b937-ac5c8ebd6c55 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:17.113057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c805b357-192d-48a0-b4ff-8d98ad93515c None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:17.126676 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0043c97f-ef66-4746-8ffe-1f1d65fc23df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:17.128178 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0043c97f-ef66-4746-8ffe-1f1d65fc23df None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:17.129004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0043c97f-ef66-4746-8ffe-1f1d65fc23df None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:17.129730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0043c97f-ef66-4746-8ffe-1f1d65fc23df None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:17.131280 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0043c97f-ef66-4746-8ffe-1f1d65fc23df None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:17.132107 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0043c97f-ef66-4746-8ffe-1f1d65fc23df None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:17.134990 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0043c97f-ef66-4746-8ffe-1f1d65fc23df None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:17.164754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 460/929] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:16:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:17.171346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0043c97f-ef66-4746-8ffe-1f1d65fc23df None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:17.181222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8403f0ac-c125-4459-9128-1873f5fbe52e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:17.182339 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8403f0ac-c125-4459-9128-1873f5fbe52e None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:17.182835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8403f0ac-c125-4459-9128-1873f5fbe52e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:17.183300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8403f0ac-c125-4459-9128-1873f5fbe52e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:17.185279 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8403f0ac-c125-4459-9128-1873f5fbe52e None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:17.189534 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8403f0ac-c125-4459-9128-1873f5fbe52e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:17.236342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 470/930] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:16:16 2022] POST /identity/v3/endpoints => generated 346 bytes in 331 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:16:17.334754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:17.342070 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0043c97f-ef66-4746-8ffe-1f1d65fc23df None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:17.404711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04d6a13-e3d3-41d8-a5f8-11a49bc1377a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:17.405823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04d6a13-e3d3-41d8-a5f8-11a49bc1377a None admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:17.406168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04d6a13-e3d3-41d8-a5f8-11a49bc1377a None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:17.406547 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04d6a13-e3d3-41d8-a5f8-11a49bc1377a None admin] PATH_INFO: `/v3/projects/895ece8b70ef44adadc3416f81a80731/users/34afa4aac10c4a018ca05a0ba34c940a/roles/47e35bb0c3b44184986f4e28032e8c17` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:17.409094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04d6a13-e3d3-41d8-a5f8-11a49bc1377a None admin] RBAC: Authorizing `identity:create_grant(project_id=895ece8b70ef44adadc3416f81a80731, user_id=34afa4aac10c4a018ca05a0ba34c940a, role_id=47e35bb0c3b44184986f4e28032e8c17)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:17.410717 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04d6a13-e3d3-41d8-a5f8-11a49bc1377a None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:17.431026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 461/931] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:16:17 2022] PUT /identity/v3/projects/895ece8b70ef44adadc3416f81a80731/users/34afa4aac10c4a018ca05a0ba34c940a/roles/47e35bb0c3b44184986f4e28032e8c17 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:16:17.680453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e6308e-d684-4a92-b4fb-a8759b67bd50 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:17.680743 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e6308e-d684-4a92-b4fb-a8759b67bd50 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:17.681025 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e6308e-d684-4a92-b4fb-a8759b67bd50 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:17.682478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 471/932] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:16:17.689558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464a91a5-61a9-4656-b316-e2551181f4de None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:17.689928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464a91a5-61a9-4656-b316-e2551181f4de None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:17.690193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464a91a5-61a9-4656-b316-e2551181f4de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:17.712277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-464a91a5-61a9-4656-b316-e2551181f4de None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:17.713618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-464a91a5-61a9-4656-b316-e2551181f4de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:17.797155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 462/933] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:17 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:18.056818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451631c3-d012-494f-9b4f-011ab6df525e None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:18.057630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451631c3-d012-494f-9b4f-011ab6df525e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:18.057630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451631c3-d012-494f-9b4f-011ab6df525e None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:18.059430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 472/934] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:18.065519 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdde41be-9414-45cb-8d6d-5b23456ab398 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:18.066362 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdde41be-9414-45cb-8d6d-5b23456ab398 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:18.066362 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdde41be-9414-45cb-8d6d-5b23456ab398 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:18.088827 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-bdde41be-9414-45cb-8d6d-5b23456ab398 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:18.092351 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-bdde41be-9414-45cb-8d6d-5b23456ab398 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:18.192133 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 463/935] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:18 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:18.317786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451631c3-d012-494f-9b4f-011ab6df525e None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:18.449344 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:18.511675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdde41be-9414-45cb-8d6d-5b23456ab398 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:18.529459 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:18.545762 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451631c3-d012-494f-9b4f-011ab6df525e None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:18.610173 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17912d1a-ef06-479e-b019-4c069433ad1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:18.611043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17912d1a-ef06-479e-b019-4c069433ad1e None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:18.611506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17912d1a-ef06-479e-b019-4c069433ad1e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:18.612017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17912d1a-ef06-479e-b019-4c069433ad1e None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:18.613125 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17912d1a-ef06-479e-b019-4c069433ad1e None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:18.615434 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17912d1a-ef06-479e-b019-4c069433ad1e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:18.634080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-17912d1a-ef06-479e-b019-4c069433ad1e None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 14:16:18.634080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:18.634080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:16:18.634080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:16:18.634080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:16:18.634080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:16:18.634080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:16:18.634080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:18.634080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 14:16:18.634080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:18.634080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:18.634080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:18.634080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:18.634080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:18.634080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:18.634080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:18.635953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:18.635953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:18.635953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:18.635953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:18.635953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:18.635953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:18.635953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:18.635953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:16:18.635953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:16:18.635953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:16:18.635953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:16:18.635953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:18.635953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:18.635953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:18.635953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:18.635953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:18.635953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:18.638700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:18.638700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:18.638700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:18.638700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:18.638700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:18.638700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:18.638700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:18.638700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:18.638700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:18.638700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:18.638700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:16:18.638700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:18.638700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 14:16:18.638700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:18.641673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 473/936] 10.222.0.64 () {56 vars in 1132 bytes} [Tue Oct 4 14:16:18 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:18.647012 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17912d1a-ef06-479e-b019-4c069433ad1e None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:18.707068 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdde41be-9414-45cb-8d6d-5b23456ab398 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:18.783804 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52192e22-f499-4f00-9d91-05fc98dfec41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:18.785719 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52192e22-f499-4f00-9d91-05fc98dfec41 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:18.786660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52192e22-f499-4f00-9d91-05fc98dfec41 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:18.787580 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52192e22-f499-4f00-9d91-05fc98dfec41 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:18.789014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52192e22-f499-4f00-9d91-05fc98dfec41 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:18.791776 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52192e22-f499-4f00-9d91-05fc98dfec41 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:18.810155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-52192e22-f499-4f00-9d91-05fc98dfec41 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 14:16:18.810155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:18.810155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:16:18.810155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:16:18.810155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:16:18.810155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:16:18.810155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:16:18.810155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:18.810155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 14:16:18.810155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:18.810155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:18.810155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:18.810155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:18.810155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:18.810155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:18.810155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:18.810155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:18.812137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:18.812137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:18.812137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:18.812137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:18.812137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:18.812137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:18.812137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:16:18.812137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:16:18.812137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:16:18.812137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:16:18.812137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:18.812137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:18.812137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:18.812137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:18.812137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:18.812137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:18.813876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:18.813876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:18.813876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:18.813876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:18.813876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:18.813876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:18.813876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:18.813876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:18.813876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:18.813876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:18.813876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:16:18.813876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:18.813876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 14:16:18.813876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:18.828922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 464/937] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Oct 4 14:16:18 2022] GET /identity/v3/services/object-store => generated 93 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:18.840050 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52192e22-f499-4f00-9d91-05fc98dfec41 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:18.904455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17912d1a-ef06-479e-b019-4c069433ad1e None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:18.988550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:19.063861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3922532c-a359-41bd-bf69-50d0adef33b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:19.067778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3922532c-a359-41bd-bf69-50d0adef33b3 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:19.068968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3922532c-a359-41bd-bf69-50d0adef33b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:19.069844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3922532c-a359-41bd-bf69-50d0adef33b3 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:19.071371 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3922532c-a359-41bd-bf69-50d0adef33b3 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:19.072214 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3922532c-a359-41bd-bf69-50d0adef33b3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:19.085920 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3922532c-a359-41bd-bf69-50d0adef33b3 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:19.104523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 474/938] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Oct 4 14:16:18 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 458 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:19.120355 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3922532c-a359-41bd-bf69-50d0adef33b3 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:19.204816 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:19.245509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52192e22-f499-4f00-9d91-05fc98dfec41 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:19.370863 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ed7d73-0d4c-46ea-a9a6-c633f9bc1959 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:19.372473 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ed7d73-0d4c-46ea-a9a6-c633f9bc1959 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:19.373154 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ed7d73-0d4c-46ea-a9a6-c633f9bc1959 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:19.373806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ed7d73-0d4c-46ea-a9a6-c633f9bc1959 None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:19.374670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ed7d73-0d4c-46ea-a9a6-c633f9bc1959 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:19.375183 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ed7d73-0d4c-46ea-a9a6-c633f9bc1959 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:19.379084 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ed7d73-0d4c-46ea-a9a6-c633f9bc1959 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:19.393958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 465/939] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Oct 4 14:16:18 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 555 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:19.406176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ed7d73-0d4c-46ea-a9a6-c633f9bc1959 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:19.438205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3922532c-a359-41bd-bf69-50d0adef33b3 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:19.482522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6337fe-f0ea-4c03-bf98-20087fe26bd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:19.483044 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6337fe-f0ea-4c03-bf98-20087fe26bd8 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:19.483325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6337fe-f0ea-4c03-bf98-20087fe26bd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:19.483663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6337fe-f0ea-4c03-bf98-20087fe26bd8 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:19.484337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6337fe-f0ea-4c03-bf98-20087fe26bd8 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:19.484631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6337fe-f0ea-4c03-bf98-20087fe26bd8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:19.486029 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6337fe-f0ea-4c03-bf98-20087fe26bd8 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:19.497490 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 475/940] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Oct 4 14:16:19 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 375 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:19.498346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6337fe-f0ea-4c03-bf98-20087fe26bd8 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:19.556360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:19.618008 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ed7d73-0d4c-46ea-a9a6-c633f9bc1959 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:19.624585 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6337fe-f0ea-4c03-bf98-20087fe26bd8 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:19.667354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b48e46-ca12-4ff8-b257-39d894fe673f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:19.667974 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b48e46-ca12-4ff8-b257-39d894fe673f None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:19.668252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b48e46-ca12-4ff8-b257-39d894fe673f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:19.668551 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b48e46-ca12-4ff8-b257-39d894fe673f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:19.669875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b48e46-ca12-4ff8-b257-39d894fe673f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:19.671403 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b48e46-ca12-4ff8-b257-39d894fe673f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:19.687278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7f1fd8-dd29-4d3d-a619-3a44f7e95afa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:19.688726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7f1fd8-dd29-4d3d-a619-3a44f7e95afa None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:19.689574 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7f1fd8-dd29-4d3d-a619-3a44f7e95afa None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:19.690392 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7f1fd8-dd29-4d3d-a619-3a44f7e95afa None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:19.691834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7f1fd8-dd29-4d3d-a619-3a44f7e95afa None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:19.692650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7f1fd8-dd29-4d3d-a619-3a44f7e95afa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:19.695437 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7f1fd8-dd29-4d3d-a619-3a44f7e95afa None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:19.701299 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 476/941] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:16:19 2022] POST /identity/v3/endpoints => generated 347 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:16:19.712508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 466/942] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Oct 4 14:16:19 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:19.718351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b48e46-ca12-4ff8-b257-39d894fe673f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:19.853874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:19.864300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b48e46-ca12-4ff8-b257-39d894fe673f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:19.907501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b956a7-59c7-4d2b-8edb-49c4c98c34d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:19.908137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b956a7-59c7-4d2b-8edb-49c4c98c34d7 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:19.908450 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b956a7-59c7-4d2b-8edb-49c4c98c34d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:19.908715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b956a7-59c7-4d2b-8edb-49c4c98c34d7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:19.909436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b956a7-59c7-4d2b-8edb-49c4c98c34d7 None admin] RBAC: Adding query filter params (interface=public, service_id=5e7334324fd54270ae6f097612b336a0, region_id=RegionOne) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:19.909745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b956a7-59c7-4d2b-8edb-49c4c98c34d7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:19.911219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b956a7-59c7-4d2b-8edb-49c4c98c34d7 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:19.918015 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 477/943] 10.222.0.64 () {56 vars in 1251 bytes} [Tue Oct 4 14:16:19 2022] GET /identity/v3/endpoints?service_id=5e7334324fd54270ae6f097612b336a0&interface=public®ion_id=RegionOne => generated 194 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:19.927922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7f1fd8-dd29-4d3d-a619-3a44f7e95afa None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:19.947396 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:20.126298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d418f57-dc12-4df8-a9d6-2dcf9740f074 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:20.127139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d418f57-dc12-4df8-a9d6-2dcf9740f074 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:20.127139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d418f57-dc12-4df8-a9d6-2dcf9740f074 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:20.128738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 478/944] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:16:20.134387 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3504a52d-cf9b-40d2-a932-5da3b5ec9503 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:20.134782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3504a52d-cf9b-40d2-a932-5da3b5ec9503 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:20.134964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3504a52d-cf9b-40d2-a932-5da3b5ec9503 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:20.139599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7f1fd8-dd29-4d3d-a619-3a44f7e95afa None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:20.155430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-3504a52d-cf9b-40d2-a932-5da3b5ec9503 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:20.156664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-3504a52d-cf9b-40d2-a932-5da3b5ec9503 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:20.195556 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53081e2-09bb-4b19-bcce-f46055826834 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:20.196201 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53081e2-09bb-4b19-bcce-f46055826834 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:20.196514 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53081e2-09bb-4b19-bcce-f46055826834 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:20.196797 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53081e2-09bb-4b19-bcce-f46055826834 None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:20.197748 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53081e2-09bb-4b19-bcce-f46055826834 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:20.199206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53081e2-09bb-4b19-bcce-f46055826834 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:20.216401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 467/945] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Oct 4 14:16:19 2022] GET /identity/v3/services => generated 1497 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:16:20.216401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe255354-9ebe-454b-a745-b8c49df2ba5f None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:20.216401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe255354-9ebe-454b-a745-b8c49df2ba5f None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:20.216401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe255354-9ebe-454b-a745-b8c49df2ba5f None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:20.217456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 468/946] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:20 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:20.220505 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a22c7c3-c541-4e61-a368-e2f47a4980eb None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:20.220842 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a22c7c3-c541-4e61-a368-e2f47a4980eb None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:20.221062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a22c7c3-c541-4e61-a368-e2f47a4980eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:20.251476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-4a22c7c3-c541-4e61-a368-e2f47a4980eb None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:20.253887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-4a22c7c3-c541-4e61-a368-e2f47a4980eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:20.295006 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 479/947] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:20 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:20.339815 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:20.350913 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 469/948] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:20 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:16:20.580947 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3504a52d-cf9b-40d2-a932-5da3b5ec9503 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:20.649445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a22c7c3-c541-4e61-a368-e2f47a4980eb None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:20.763006 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:20.772169 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3504a52d-cf9b-40d2-a932-5da3b5ec9503 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:20.816259 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a22c7c3-c541-4e61-a368-e2f47a4980eb None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:20.823552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e05682-1204-4ad4-99a2-c8bde448f0c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:20.823552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e05682-1204-4ad4-99a2-c8bde448f0c8 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:20.823552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e05682-1204-4ad4-99a2-c8bde448f0c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:20.824093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e05682-1204-4ad4-99a2-c8bde448f0c8 None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:20.824716 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e05682-1204-4ad4-99a2-c8bde448f0c8 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:20.827224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e05682-1204-4ad4-99a2-c8bde448f0c8 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:20.830220 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: WARNING keystone.server.flask.common [None req-a5e05682-1204-4ad4-99a2-c8bde448f0c8 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 04 14:16:20.852786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.sql.core [None req-a5e05682-1204-4ad4-99a2-c8bde448f0c8 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 04 14:16:20.852786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 04 14:16:20.852786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [parameters: {'id': 'f86d334dccac460d971d7708a91ec985', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 04 14:16:20.852786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81461) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 04 14:16:20.863325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-a5e05682-1204-4ad4-99a2-c8bde448f0c8 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id f86d334dccac460d971d7708a91ec985.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id f86d334dccac460d971d7708a91ec985. Oct 04 14:16:20.863325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:20.863325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 14:16:20.863325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 14:16:20.863325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 14:16:20.863325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 14:16:20.863325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 14:16:20.863325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 14:16:20.863325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 14:16:20.863325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application conn.query(q) Oct 04 14:16:20.863325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 14:16:20.863325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 14:16:20.863325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application result.read() Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 04 14:16:20.864936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 04 14:16:20.866798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 14:16:20.866798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application next(self.gen) Oct 04 14:16:20.866798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 04 14:16:20.866798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application yield resource Oct 04 14:16:20.866798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 14:16:20.866798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application next(self.gen) Oct 04 14:16:20.866798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 04 14:16:20.866798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self.session.rollback() Oct 04 14:16:20.866798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 04 14:16:20.866798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self.force_reraise() Oct 04 14:16:20.866798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 04 14:16:20.866798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise self.value Oct 04 14:16:20.866798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 04 14:16:20.866798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 04 14:16:20.866798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 04 14:16:20.866798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application session.commit() Oct 04 14:16:20.866798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 04 14:16:20.868491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 04 14:16:20.868491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 04 14:16:20.868491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._prepare_impl() Oct 04 14:16:20.868491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 04 14:16:20.868491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self.session.flush() Oct 04 14:16:20.868491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 04 14:16:20.868491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._flush(objects) Oct 04 14:16:20.868491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 04 14:16:20.868491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 04 14:16:20.868491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 04 14:16:20.868491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application compat.raise_( Oct 04 14:16:20.868491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 14:16:20.868491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception Oct 04 14:16:20.868491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 04 14:16:20.868491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application flush_context.execute() Oct 04 14:16:20.868491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 04 14:16:20.868491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rec.execute(self) Oct 04 14:16:20.870589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 04 14:16:20.870589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 04 14:16:20.870589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 04 14:16:20.870589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 04 14:16:20.870589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 04 14:16:20.870589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 04 14:16:20.870589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 04 14:16:20.870589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 04 14:16:20.870589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 04 14:16:20.870589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 04 14:16:20.870589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 04 14:16:20.870589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 04 14:16:20.870589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 04 14:16:20.870589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 04 14:16:20.870589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 04 14:16:20.877508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 04 14:16:20.877508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 14:16:20.877508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception Oct 04 14:16:20.877508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 14:16:20.877508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 14:16:20.877508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 14:16:20.877508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 14:16:20.877508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 14:16:20.877508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 14:16:20.877508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 14:16:20.877508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application conn.query(q) Oct 04 14:16:20.877508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 14:16:20.877508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 14:16:20.877508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 14:16:20.877508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application result.read() Oct 04 14:16:20.877508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 14:16:20.877508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 14:16:20.882206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 14:16:20.882206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 14:16:20.882206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 14:16:20.882206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 14:16:20.882206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 14:16:20.882206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 14:16:20.882206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 04 14:16:20.882206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 04 14:16:20.882206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [parameters: {'id': 'f86d334dccac460d971d7708a91ec985', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 04 14:16:20.882206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 04 14:16:20.882206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:20.882206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:20.882206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:20.882206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:20.882206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 04 14:16:20.884843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 04 14:16:20.884843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 04 14:16:20.884843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 04 14:16:20.884843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 04 14:16:20.884843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:20.884843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:20.884843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:20.884843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:20.884843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:20.884843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:20.884843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:20.884843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:20.884843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:20.884843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:20.884843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:20.884843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:20.884843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:20.893310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:20.893310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 04 14:16:20.893310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 04 14:16:20.893310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:20.893310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:20.893310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 04 14:16:20.893310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 04 14:16:20.893310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id f86d334dccac460d971d7708a91ec985. Oct 04 14:16:20.893310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:20.893310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 480/949] 10.222.0.64 () {60 vars in 1140 bytes} [Tue Oct 4 14:16:20 2022] POST /identity/v3/projects => generated 262 bytes in 288 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:20.893310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e05682-1204-4ad4-99a2-c8bde448f0c8 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:20.929984 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8279a60e-3e6f-4ae4-b950-c4400a0c97fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:20.931455 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8279a60e-3e6f-4ae4-b950-c4400a0c97fc None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:20.932210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8279a60e-3e6f-4ae4-b950-c4400a0c97fc None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:20.932962 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8279a60e-3e6f-4ae4-b950-c4400a0c97fc None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:20.961949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8279a60e-3e6f-4ae4-b950-c4400a0c97fc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:16:20.976516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-8279a60e-3e6f-4ae4-b950-c4400a0c97fc None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 14:16:20.976516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:20.976516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:20.976516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:20.976516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:20.976516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:20.976516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:20.976516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:20.976516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:20.976516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:20.976516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:20.976516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:20.976516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:16:20.976516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:16:20.976516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:16:20.976516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:16:20.978449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:16:20.978449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:16:20.978449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:16:20.978449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:16:20.978449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:20.978449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:20.978449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:20.978449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:20.978449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:20.978449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:20.978449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:20.978449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:20.978449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:20.978449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:20.978449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:20.978449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:20.980388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:20.980388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:20.980388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:20.980388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:20.980388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:16:20.980388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:16:20.980388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:16:20.980388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:16:20.980388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:16:20.980388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:16:20.980388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 14:16:20.980388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:20.985782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 470/950] 10.222.0.64 () {56 vars in 1120 bytes} [Tue Oct 4 14:16:20 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 336 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:20.992355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8279a60e-3e6f-4ae4-b950-c4400a0c97fc None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:21.056462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e05682-1204-4ad4-99a2-c8bde448f0c8 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:21.079762 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:21.124396 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4489a35-9098-4f44-b511-432e7a29df61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:21.125478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4489a35-9098-4f44-b511-432e7a29df61 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:21.125946 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4489a35-9098-4f44-b511-432e7a29df61 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:21.126411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4489a35-9098-4f44-b511-432e7a29df61 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:21.135164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8279a60e-3e6f-4ae4-b950-c4400a0c97fc None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:21.140415 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4489a35-9098-4f44-b511-432e7a29df61 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:21.143605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4489a35-9098-4f44-b511-432e7a29df61 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:21.169289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-f4489a35-9098-4f44-b511-432e7a29df61 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:21.169289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:21.169289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:21.169289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:21.169289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:21.169289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:21.169289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:21.169289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:21.169289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:21.169289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:21.169289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:21.169289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:21.169289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:16:21.169289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:16:21.169289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:16:21.170659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:16:21.170659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:21.170659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:21.170659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:21.170659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:21.170659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:21.170659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:21.170659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:21.170659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:21.170659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:21.170659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:21.170659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:21.170659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:21.170659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:21.170659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:21.170659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:21.170659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:21.171666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:16:21.171666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:16:21.171666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:16:21.171666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:16:21.171666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:16:21.171666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:16:21.171666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:21.171666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:21.179180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 481/951] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:16:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:21.181236 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4489a35-9098-4f44-b511-432e7a29df61 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:21.183487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653958c0-72e4-4133-b17a-aad98cca40be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:21.184818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653958c0-72e4-4133-b17a-aad98cca40be None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:21.185437 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653958c0-72e4-4133-b17a-aad98cca40be None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:21.185865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653958c0-72e4-4133-b17a-aad98cca40be None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:21.186843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653958c0-72e4-4133-b17a-aad98cca40be None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:21.187301 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653958c0-72e4-4133-b17a-aad98cca40be None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:21.189046 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653958c0-72e4-4133-b17a-aad98cca40be None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:21.199255 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 471/952] 10.222.0.64 () {56 vars in 1115 bytes} [Tue Oct 4 14:16:20 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:21.206677 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:21.207028 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653958c0-72e4-4133-b17a-aad98cca40be None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:21.341874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4489a35-9098-4f44-b511-432e7a29df61 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:21.342709 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653958c0-72e4-4133-b17a-aad98cca40be None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:21.372922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:21.383954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770cf0f9-d203-4147-ae08-9313fb1cd110 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:21.386967 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770cf0f9-d203-4147-ae08-9313fb1cd110 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:21.387252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770cf0f9-d203-4147-ae08-9313fb1cd110 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:21.387568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770cf0f9-d203-4147-ae08-9313fb1cd110 None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:21.388411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770cf0f9-d203-4147-ae08-9313fb1cd110 None admin] RBAC: Adding query filter params (name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:21.388718 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770cf0f9-d203-4147-ae08-9313fb1cd110 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:21.390237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770cf0f9-d203-4147-ae08-9313fb1cd110 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:21.391284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1809aeb-9c9c-4dac-8f4e-362faaac4b0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:21.392471 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1809aeb-9c9c-4dac-8f4e-362faaac4b0d None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:21.392989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1809aeb-9c9c-4dac-8f4e-362faaac4b0d None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:21.393432 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1809aeb-9c9c-4dac-8f4e-362faaac4b0d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:21.408742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1809aeb-9c9c-4dac-8f4e-362faaac4b0d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:21.410509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1809aeb-9c9c-4dac-8f4e-362faaac4b0d None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:21.420492 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 482/953] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Oct 4 14:16:21 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:21.429018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-c1809aeb-9c9c-4dac-8f4e-362faaac4b0d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:21.429018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:21.429018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:21.429018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:21.429018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:21.429018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:21.429018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:21.429018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:21.429018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:21.429018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:21.429018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:21.429018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:21.429018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:21.429018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:21.429018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:21.429018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:21.430111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:21.430111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:21.430111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:21.430111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:21.430111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:21.430111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:21.430111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:21.430111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:21.430111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:21.430111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:21.430111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:21.430111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:21.430111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:21.430111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:21.430111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:21.430111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:21.430111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:21.431125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:21.431125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:21.431125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:21.433645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 472/954] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:21.440994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770cf0f9-d203-4147-ae08-9313fb1cd110 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:21.536940 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:21.581660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770cf0f9-d203-4147-ae08-9313fb1cd110 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:21.627691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39ffadf-7c36-4a4a-a0cc-f89343327621 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:21.628358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39ffadf-7c36-4a4a-a0cc-f89343327621 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:21.628644 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39ffadf-7c36-4a4a-a0cc-f89343327621 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:21.628918 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39ffadf-7c36-4a4a-a0cc-f89343327621 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:21.629586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39ffadf-7c36-4a4a-a0cc-f89343327621 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:21.629889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39ffadf-7c36-4a4a-a0cc-f89343327621 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:21.631344 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39ffadf-7c36-4a4a-a0cc-f89343327621 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:21.645665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 483/955] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:21.651273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1809aeb-9c9c-4dac-8f4e-362faaac4b0d None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:21.776828 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1809aeb-9c9c-4dac-8f4e-362faaac4b0d None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:21.818524 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f87a06f-93ec-43b8-a6ff-a3fb1f708289 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:21.819293 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f87a06f-93ec-43b8-a6ff-a3fb1f708289 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:21.819584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f87a06f-93ec-43b8-a6ff-a3fb1f708289 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:21.819865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f87a06f-93ec-43b8-a6ff-a3fb1f708289 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:21.830069 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f87a06f-93ec-43b8-a6ff-a3fb1f708289 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:21.831616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f87a06f-93ec-43b8-a6ff-a3fb1f708289 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:21.845886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-6f87a06f-93ec-43b8-a6ff-a3fb1f708289 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 14:16:21.845886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:21.845886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:21.845886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:21.845886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:21.845886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:21.845886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:21.845886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:21.845886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:21.845886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:21.845886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:21.845886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:21.845886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:16:21.845886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:16:21.845886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:16:21.845886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:21.846939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:21.847949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:21.847949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:21.847949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:16:21.847949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:16:21.847949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:16:21.847949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:16:21.847949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:16:21.847949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:16:21.847949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 14:16:21.847949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:21.847949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 473/956] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Oct 4 14:16:21 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:21.855269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39ffadf-7c36-4a4a-a0cc-f89343327621 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:21.981975 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39ffadf-7c36-4a4a-a0cc-f89343327621 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:22.014908 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:22.026007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baca6c98-9a06-4c33-aef0-f8b7b384fb7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:22.027033 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baca6c98-9a06-4c33-aef0-f8b7b384fb7a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:22.027575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baca6c98-9a06-4c33-aef0-f8b7b384fb7a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:22.028924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baca6c98-9a06-4c33-aef0-f8b7b384fb7a None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:22.030289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baca6c98-9a06-4c33-aef0-f8b7b384fb7a None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:22.030844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baca6c98-9a06-4c33-aef0-f8b7b384fb7a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:22.032559 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baca6c98-9a06-4c33-aef0-f8b7b384fb7a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:22.054195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 484/957] 10.222.0.64 () {56 vars in 1149 bytes} [Tue Oct 4 14:16:21 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:22.061471 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f87a06f-93ec-43b8-a6ff-a3fb1f708289 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:22.104815 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:22.252376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f87a06f-93ec-43b8-a6ff-a3fb1f708289 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:22.326564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccb47a7-2f8c-4415-babe-3a6bebb4b591 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:22.327588 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccb47a7-2f8c-4415-babe-3a6bebb4b591 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:22.328092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccb47a7-2f8c-4415-babe-3a6bebb4b591 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:22.328589 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccb47a7-2f8c-4415-babe-3a6bebb4b591 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:22.404348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5595550d-ce0c-46b1-af34-6e0a4e1e327f None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:22.404914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5595550d-ce0c-46b1-af34-6e0a4e1e327f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:22.405371 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5595550d-ce0c-46b1-af34-6e0a4e1e327f None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:22.407816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 485/958] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:16:22.413297 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a307f315-3d14-4d0a-965f-8da9b0a32914 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:22.413634 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a307f315-3d14-4d0a-965f-8da9b0a32914 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:22.413848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a307f315-3d14-4d0a-965f-8da9b0a32914 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:22.421198 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccb47a7-2f8c-4415-babe-3a6bebb4b591 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:22.423457 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccb47a7-2f8c-4415-babe-3a6bebb4b591 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:22.436768 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-a307f315-3d14-4d0a-965f-8da9b0a32914 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:22.438370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a307f315-3d14-4d0a-965f-8da9b0a32914 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:22.463282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:22.554688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 474/959] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:16:22 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 489 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 14:16:22.554885 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 486/960] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:22 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:16:22.556040 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccb47a7-2f8c-4415-babe-3a6bebb4b591 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:22.679404 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccb47a7-2f8c-4415-babe-3a6bebb4b591 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:22.720219 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79431590-0d20-4717-a9d4-1465496f6adc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:22.721356 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79431590-0d20-4717-a9d4-1465496f6adc None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:22.721850 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79431590-0d20-4717-a9d4-1465496f6adc None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:22.722341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79431590-0d20-4717-a9d4-1465496f6adc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:22.725037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:22.737133 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79431590-0d20-4717-a9d4-1465496f6adc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:22.738672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79431590-0d20-4717-a9d4-1465496f6adc None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:22.765678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-79431590-0d20-4717-a9d4-1465496f6adc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:22.765678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:22.765678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:22.765678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:22.765678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:22.765678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:22.765678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:22.765678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:22.765678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:22.765678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:22.765678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:22.765678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:22.765678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:22.765678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:22.765678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:22.765678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:22.766735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:22.766735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:22.766735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:22.766735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:22.766735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:22.766735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:22.766735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:22.766735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:22.766735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:22.766735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:22.766735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:22.766735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:22.766735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:22.766735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:22.766735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:22.766735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:22.766735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:22.767643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:22.767643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:22.767643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:22.768217 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 475/961] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:22.773619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a307f315-3d14-4d0a-965f-8da9b0a32914 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:22.864684 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79431590-0d20-4717-a9d4-1465496f6adc None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:22.896885 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a307f315-3d14-4d0a-965f-8da9b0a32914 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:22.918341 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:22.940418 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c582f968-ba6b-4e05-9261-7078d7973177 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:22.941046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c582f968-ba6b-4e05-9261-7078d7973177 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:22.941322 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c582f968-ba6b-4e05-9261-7078d7973177 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:22.941591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c582f968-ba6b-4e05-9261-7078d7973177 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:22.943336 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c582f968-ba6b-4e05-9261-7078d7973177 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:22.943336 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c582f968-ba6b-4e05-9261-7078d7973177 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:22.943918 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c582f968-ba6b-4e05-9261-7078d7973177 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:22.962398 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 487/962] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:22.963434 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c582f968-ba6b-4e05-9261-7078d7973177 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:23.073500 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79431590-0d20-4717-a9d4-1465496f6adc None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:23.090207 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c582f968-ba6b-4e05-9261-7078d7973177 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:23.134940 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a008749-fe6f-4fa9-87df-b31a742426b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:23.135787 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a008749-fe6f-4fa9-87df-b31a742426b5 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:23.136247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a008749-fe6f-4fa9-87df-b31a742426b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:23.136676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a008749-fe6f-4fa9-87df-b31a742426b5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:23.140536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc393cf-0517-4209-bd89-a58d734997c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:23.141976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc393cf-0517-4209-bd89-a58d734997c8 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:23.142699 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc393cf-0517-4209-bd89-a58d734997c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:23.143413 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc393cf-0517-4209-bd89-a58d734997c8 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:23.144773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc393cf-0517-4209-bd89-a58d734997c8 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:23.147702 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc393cf-0517-4209-bd89-a58d734997c8 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:23.149289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a008749-fe6f-4fa9-87df-b31a742426b5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:23.150882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a008749-fe6f-4fa9-87df-b31a742426b5 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:23.164777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-1a008749-fe6f-4fa9-87df-b31a742426b5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:23.164777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:23.164777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:23.164777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:23.164777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:23.164777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:23.164777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:23.164777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:23.164777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:23.164777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:23.164777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:23.164777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:23.164777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:16:23.164777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:16:23.164777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:16:23.166857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:16:23.166857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:23.166857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:23.166857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:23.166857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:23.166857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:23.166857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:23.166857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:23.166857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:23.166857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:23.166857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:23.166857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:23.166857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:23.166857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:23.166857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:23.166857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:23.166857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:23.168867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:16:23.168867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:16:23.168867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:16:23.168867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:16:23.168867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:16:23.168867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:16:23.168867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:23.168867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:23.168867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 488/963] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:16:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:23.169826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-ddc393cf-0517-4209-bd89-a58d734997c8 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 14:16:23.169826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:23.169826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:16:23.169826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:16:23.169826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:16:23.169826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:16:23.169826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:16:23.169826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:23.169826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 14:16:23.169826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:23.169826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:23.169826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:23.169826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:23.169826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:23.169826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:23.169826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:23.169826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:23.171672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:23.171672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:23.171672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:23.171672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:23.171672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:23.171672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:23.171672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:16:23.171672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:16:23.171672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:16:23.171672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:16:23.171672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:23.171672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:23.171672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:23.171672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:23.171672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:23.171672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:23.174955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:23.174955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:23.174955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:23.174955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:23.174955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:23.174955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:23.174955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:23.174955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:23.174955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:23.174955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:23.174955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:16:23.174955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:23.174955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 14:16:23.174955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:23.178166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a008749-fe6f-4fa9-87df-b31a742426b5 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:23.178437 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 476/964] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:16:22 2022] GET /identity/v3/services/compute => generated 88 bytes in 308 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:23.178685 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc393cf-0517-4209-bd89-a58d734997c8 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:23.222607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:23.309905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a008749-fe6f-4fa9-87df-b31a742426b5 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:23.314897 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:23.383207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc393cf-0517-4209-bd89-a58d734997c8 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:23.404908 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c117acf-8d05-4658-836d-2b015e6a25aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:23.405746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c117acf-8d05-4658-836d-2b015e6a25aa None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:23.405746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c117acf-8d05-4658-836d-2b015e6a25aa None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:23.405746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c117acf-8d05-4658-836d-2b015e6a25aa None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:23.406379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c117acf-8d05-4658-836d-2b015e6a25aa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:23.406692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c117acf-8d05-4658-836d-2b015e6a25aa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:23.408155 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c117acf-8d05-4658-836d-2b015e6a25aa None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:23.426174 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 489/965] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:16:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:23.432130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c117acf-8d05-4658-836d-2b015e6a25aa None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:23.477130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1818a147-10bf-40fc-b3af-e9fac5897b04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:23.480273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1818a147-10bf-40fc-b3af-e9fac5897b04 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:23.481188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1818a147-10bf-40fc-b3af-e9fac5897b04 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:23.481905 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1818a147-10bf-40fc-b3af-e9fac5897b04 None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:23.483683 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1818a147-10bf-40fc-b3af-e9fac5897b04 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:23.485175 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1818a147-10bf-40fc-b3af-e9fac5897b04 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:23.486211 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1818a147-10bf-40fc-b3af-e9fac5897b04 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:23.499648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 477/966] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Oct 4 14:16:23 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:23.509138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1818a147-10bf-40fc-b3af-e9fac5897b04 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:23.568748 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:23.584859 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c117acf-8d05-4658-836d-2b015e6a25aa None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:23.635256 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8471a1-71af-4c18-bf16-b4757914a5ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:23.636629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8471a1-71af-4c18-bf16-b4757914a5ab None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:23.637037 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8471a1-71af-4c18-bf16-b4757914a5ab None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:23.637830 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8471a1-71af-4c18-bf16-b4757914a5ab None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:23.639008 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8471a1-71af-4c18-bf16-b4757914a5ab None admin] RBAC: Adding query filter params (role.id=47e35bb0c3b44184986f4e28032e8c17, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=34afa4aac10c4a018ca05a0ba34c940a) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:23.639538 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8471a1-71af-4c18-bf16-b4757914a5ab None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:23.641344 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8471a1-71af-4c18-bf16-b4757914a5ab None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:23.652558 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 490/967] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:16:23 2022] GET /identity/v3/role_assignments?user.id=34afa4aac10c4a018ca05a0ba34c940a&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=47e35bb0c3b44184986f4e28032e8c17 => generated 608 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:23.672039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1818a147-10bf-40fc-b3af-e9fac5897b04 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:23.692634 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:23.721274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f8315f-6531-4ec5-a132-cf77e7e1af79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:23.722048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f8315f-6531-4ec5-a132-cf77e7e1af79 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:23.722334 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f8315f-6531-4ec5-a132-cf77e7e1af79 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:23.722612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f8315f-6531-4ec5-a132-cf77e7e1af79 None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:23.723313 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f8315f-6531-4ec5-a132-cf77e7e1af79 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:23.723627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f8315f-6531-4ec5-a132-cf77e7e1af79 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:23.725226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f8315f-6531-4ec5-a132-cf77e7e1af79 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:23.736568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 478/968] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Oct 4 14:16:23 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:23.854819 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfe091b-27d4-405d-b162-1d27c9adaab7 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:23.855092 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfe091b-27d4-405d-b162-1d27c9adaab7 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:23.855516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfe091b-27d4-405d-b162-1d27c9adaab7 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:23.857356 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 491/969] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:23.866412 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a31f03-5c63-491c-b23b-270dfe91eadf None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:23.866724 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a31f03-5c63-491c-b23b-270dfe91eadf None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:23.867074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a31f03-5c63-491c-b23b-270dfe91eadf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:23.888973 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-90a31f03-5c63-491c-b23b-270dfe91eadf None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:23.890265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-90a31f03-5c63-491c-b23b-270dfe91eadf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:23.982530 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 479/970] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:23 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:24.100787 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf4bb6-8071-4749-9368-9ce23ae9bd8d None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:24.101077 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf4bb6-8071-4749-9368-9ce23ae9bd8d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:24.101402 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf4bb6-8071-4749-9368-9ce23ae9bd8d None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:24.103033 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 492/971] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:16:24.109595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9df764b-d589-4929-a831-5ea0628b6999 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:24.110024 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9df764b-d589-4929-a831-5ea0628b6999 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:24.110175 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9df764b-d589-4929-a831-5ea0628b6999 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:24.131833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-b9df764b-d589-4929-a831-5ea0628b6999 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:24.133146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-b9df764b-d589-4929-a831-5ea0628b6999 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:24.214666 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 480/972] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:24 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:24.482083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecf4bb6-8071-4749-9368-9ce23ae9bd8d None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:24.506291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9df764b-d589-4929-a831-5ea0628b6999 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:24.556874 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:24.636022 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9df764b-d589-4929-a831-5ea0628b6999 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:24.685500 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ce27b5-27f8-46bc-86a5-e99fdd549a49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:24.686284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ce27b5-27f8-46bc-86a5-e99fdd549a49 None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:24.686566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ce27b5-27f8-46bc-86a5-e99fdd549a49 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:24.686841 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ce27b5-27f8-46bc-86a5-e99fdd549a49 None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:24.687504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ce27b5-27f8-46bc-86a5-e99fdd549a49 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:24.689034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ce27b5-27f8-46bc-86a5-e99fdd549a49 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:24.691828 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: WARNING keystone.server.flask.common [None req-b0ce27b5-27f8-46bc-86a5-e99fdd549a49 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 04 14:16:24.708029 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.sql.core [None req-b0ce27b5-27f8-46bc-86a5-e99fdd549a49 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 04 14:16:24.708029 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 04 14:16:24.708029 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [parameters: {'user_id': '3b45f6c456b84d9daee06985f8cc413c', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 04 14:16:24.708029 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81460) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 04 14:16:24.711841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecf4bb6-8071-4749-9368-9ce23ae9bd8d None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:24.728706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-b0ce27b5-27f8-46bc-86a5-e99fdd549a49 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 04 14:16:24.728706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:24.728706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 14:16:24.728706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 14:16:24.728706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 14:16:24.728706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 14:16:24.728706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 14:16:24.728706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 14:16:24.728706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 14:16:24.728706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application conn.query(q) Oct 04 14:16:24.728706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 14:16:24.728706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 14:16:24.728706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 14:16:24.728706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application result.read() Oct 04 14:16:24.728706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 14:16:24.730760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application next(self.gen) Oct 04 14:16:24.733660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 04 14:16:24.733660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application yield resource Oct 04 14:16:24.733660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 14:16:24.733660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application next(self.gen) Oct 04 14:16:24.733660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 04 14:16:24.733660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self.session.rollback() Oct 04 14:16:24.733660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 04 14:16:24.733660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self.force_reraise() Oct 04 14:16:24.733660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 04 14:16:24.733660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise self.value Oct 04 14:16:24.733660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 04 14:16:24.733660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 04 14:16:24.733660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 04 14:16:24.733660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application session.commit() Oct 04 14:16:24.733660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 04 14:16:24.733660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 04 14:16:24.733660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 04 14:16:24.735493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._prepare_impl() Oct 04 14:16:24.735493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 04 14:16:24.735493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self.session.flush() Oct 04 14:16:24.735493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 04 14:16:24.735493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._flush(objects) Oct 04 14:16:24.735493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 04 14:16:24.735493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 04 14:16:24.735493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 04 14:16:24.735493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application compat.raise_( Oct 04 14:16:24.735493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 14:16:24.735493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception Oct 04 14:16:24.735493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 04 14:16:24.735493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application flush_context.execute() Oct 04 14:16:24.735493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 04 14:16:24.735493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rec.execute(self) Oct 04 14:16:24.735493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 04 14:16:24.735493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 04 14:16:24.737706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 04 14:16:24.737706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 04 14:16:24.737706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 04 14:16:24.737706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 04 14:16:24.737706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 04 14:16:24.737706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 04 14:16:24.737706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 04 14:16:24.737706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 04 14:16:24.737706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 04 14:16:24.737706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 04 14:16:24.737706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 04 14:16:24.737706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 04 14:16:24.737706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 04 14:16:24.737706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 04 14:16:24.737706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 14:16:24.737706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception Oct 04 14:16:24.739620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 14:16:24.739620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 14:16:24.739620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 14:16:24.739620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 14:16:24.739620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 14:16:24.739620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 14:16:24.739620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 14:16:24.739620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application conn.query(q) Oct 04 14:16:24.739620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 14:16:24.739620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 14:16:24.739620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 14:16:24.739620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application result.read() Oct 04 14:16:24.739620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 14:16:24.739620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 14:16:24.739620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 14:16:24.739620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 14:16:24.744811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 14:16:24.744811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 14:16:24.744811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 14:16:24.744811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 14:16:24.744811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 04 14:16:24.744811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 04 14:16:24.744811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [parameters: {'user_id': '3b45f6c456b84d9daee06985f8cc413c', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 04 14:16:24.744811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 04 14:16:24.744811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:24.744811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:24.744811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:24.744811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:24.744811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:24.744811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:24.744811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:24.748001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:24.748001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:24.748001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:24.748001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:24.748001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:24.748001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:24.748001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:24.748001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 04 14:16:24.748001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 04 14:16:24.748001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:24.748001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:24.748001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:16:24.748001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:24.748001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:16:24.748001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:24.748001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 04 14:16:24.748001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 04 14:16:24.749958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 04 14:16:24.749958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 04 14:16:24.749958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 04 14:16:24.749958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 04 14:16:24.749958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 04 14:16:24.749958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:24.749958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 481/973] 10.222.0.64 () {60 vars in 1131 bytes} [Tue Oct 4 14:16:24 2022] POST /identity/v3/users => generated 160 bytes in 225 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:24.749958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ce27b5-27f8-46bc-86a5-e99fdd549a49 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:24.819158 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47aa85bf-1b0b-4c27-b244-a0f0beb291c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:24.821481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47aa85bf-1b0b-4c27-b244-a0f0beb291c8 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:24.822455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47aa85bf-1b0b-4c27-b244-a0f0beb291c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:24.823305 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47aa85bf-1b0b-4c27-b244-a0f0beb291c8 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:24.825701 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47aa85bf-1b0b-4c27-b244-a0f0beb291c8 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:24.829743 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47aa85bf-1b0b-4c27-b244-a0f0beb291c8 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:24.855759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-47aa85bf-1b0b-4c27-b244-a0f0beb291c8 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 14:16:24.855759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:24.855759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:16:24.855759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:16:24.855759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:16:24.855759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:16:24.855759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:16:24.855759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:24.855759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 14:16:24.855759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:24.855759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:24.855759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:24.855759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:24.855759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:24.855759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:24.855759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:24.855759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:24.857936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:24.857936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:24.857936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:24.857936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:24.857936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:24.857936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:24.857936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:16:24.857936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:16:24.857936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:16:24.857936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:16:24.857936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:24.857936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:24.857936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:24.857936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:24.857936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:24.857936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:24.859834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:24.859834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:24.859834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:24.859834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:24.859834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:24.859834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:24.859834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:24.859834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:24.859834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:24.859834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:24.859834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:16:24.859834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:24.859834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 14:16:24.859834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:24.872648 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 493/974] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Oct 4 14:16:24 2022] GET /identity/v3/services/object-store => generated 93 bytes in 388 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:24.875156 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47aa85bf-1b0b-4c27-b244-a0f0beb291c8 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:24.914591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:24.933598 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:24.943742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ce27b5-27f8-46bc-86a5-e99fdd549a49 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:25.008898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edac1b1e-1241-42f2-aecf-7cae97594cd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:25.009816 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edac1b1e-1241-42f2-aecf-7cae97594cd4 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:25.010294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edac1b1e-1241-42f2-aecf-7cae97594cd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:25.010695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edac1b1e-1241-42f2-aecf-7cae97594cd4 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:25.014643 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47aa85bf-1b0b-4c27-b244-a0f0beb291c8 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:25.024360 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edac1b1e-1241-42f2-aecf-7cae97594cd4 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:25.027270 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edac1b1e-1241-42f2-aecf-7cae97594cd4 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:25.050149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-edac1b1e-1241-42f2-aecf-7cae97594cd4 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 14:16:25.050149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:25.050149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:25.050149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:25.050149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:25.050149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:25.050149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:25.050149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:25.050149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:25.050149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:25.050149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:25.050149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:25.050149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:16:25.050149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:16:25.050149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:16:25.050149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:25.051210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:25.052169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:25.052169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:25.052169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:16:25.052169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:16:25.052169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:16:25.052169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:16:25.052169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:16:25.052169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:16:25.052169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 14:16:25.052169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:25.052785 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 482/975] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:16:24 2022] GET /identity/v3/users/cinder => generated 84 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:25.060001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edac1b1e-1241-42f2-aecf-7cae97594cd4 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:25.070223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab15a21-6b45-4be2-898a-bf6f1c34cab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:25.071302 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab15a21-6b45-4be2-898a-bf6f1c34cab0 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:25.071782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab15a21-6b45-4be2-898a-bf6f1c34cab0 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:25.072304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab15a21-6b45-4be2-898a-bf6f1c34cab0 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:25.073168 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab15a21-6b45-4be2-898a-bf6f1c34cab0 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:25.073660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab15a21-6b45-4be2-898a-bf6f1c34cab0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:25.076002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab15a21-6b45-4be2-898a-bf6f1c34cab0 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:25.082347 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 494/976] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Oct 4 14:16:24 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:25.093223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab15a21-6b45-4be2-898a-bf6f1c34cab0 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:25.224093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:25.242630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab15a21-6b45-4be2-898a-bf6f1c34cab0 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:25.244751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edac1b1e-1241-42f2-aecf-7cae97594cd4 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:25.291784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb676f96-8983-49b9-a4a3-839c757580ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:25.292775 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb676f96-8983-49b9-a4a3-839c757580ff None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:25.293383 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb676f96-8983-49b9-a4a3-839c757580ff None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:25.294518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb676f96-8983-49b9-a4a3-839c757580ff None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:25.295485 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb676f96-8983-49b9-a4a3-839c757580ff None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:25.296015 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb676f96-8983-49b9-a4a3-839c757580ff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:25.297112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37c0214-d560-4e0f-8178-b4e5b5100041 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:25.297826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37c0214-d560-4e0f-8178-b4e5b5100041 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:25.298179 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37c0214-d560-4e0f-8178-b4e5b5100041 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:25.298521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37c0214-d560-4e0f-8178-b4e5b5100041 None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:25.298775 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb676f96-8983-49b9-a4a3-839c757580ff None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:25.299378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37c0214-d560-4e0f-8178-b4e5b5100041 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:25.299754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37c0214-d560-4e0f-8178-b4e5b5100041 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:25.302496 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37c0214-d560-4e0f-8178-b4e5b5100041 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:25.306180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 495/977] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Oct 4 14:16:25 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:25.315786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb676f96-8983-49b9-a4a3-839c757580ff None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:25.353215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 483/978] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Oct 4 14:16:25 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:25.533336 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb676f96-8983-49b9-a4a3-839c757580ff None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:25.605043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc911f27-de5b-4b14-846f-0738e1f222d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:25.606038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc911f27-de5b-4b14-846f-0738e1f222d5 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:25.606503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc911f27-de5b-4b14-846f-0738e1f222d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:25.606979 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc911f27-de5b-4b14-846f-0738e1f222d5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:25.608416 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc911f27-de5b-4b14-846f-0738e1f222d5 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:25.611038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc911f27-de5b-4b14-846f-0738e1f222d5 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:25.647608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 496/979] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:16:25 2022] POST /identity/v3/endpoints => generated 349 bytes in 333 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:16:26.208152 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1316516-181f-49f6-b110-13f5a2dcd862 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:26.208948 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1316516-181f-49f6-b110-13f5a2dcd862 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:26.208948 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1316516-181f-49f6-b110-13f5a2dcd862 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:26.210137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 484/980] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:16:26.216919 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b2515-c1cd-44f1-8be8-e33c47eb01de None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:26.217182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b2515-c1cd-44f1-8be8-e33c47eb01de None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:26.217456 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b2515-c1cd-44f1-8be8-e33c47eb01de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:26.239262 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-028b2515-c1cd-44f1-8be8-e33c47eb01de None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:26.240521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-028b2515-c1cd-44f1-8be8-e33c47eb01de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:26.243272 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:26.269441 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88ac534-205e-4d8e-91f0-dab4df68015e None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:26.270285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88ac534-205e-4d8e-91f0-dab4df68015e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:26.271854 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88ac534-205e-4d8e-91f0-dab4df68015e None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:26.276093 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 485/981] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:26 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:26.283935 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b22c1b-e382-4028-a6e4-91dacda3c55e None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:26.285391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b22c1b-e382-4028-a6e4-91dacda3c55e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:26.286252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b22c1b-e382-4028-a6e4-91dacda3c55e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:26.329266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-33b22c1b-e382-4028-a6e4-91dacda3c55e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:26.333565 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-33b22c1b-e382-4028-a6e4-91dacda3c55e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:26.338478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 497/982] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:26 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:16:26.361599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:26.475543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 486/983] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:26 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:26.622891 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028b2515-c1cd-44f1-8be8-e33c47eb01de None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:26.753402 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028b2515-c1cd-44f1-8be8-e33c47eb01de None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:26.757881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b22c1b-e382-4028-a6e4-91dacda3c55e None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:26.801259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7095e93-241f-4db9-82a6-1f1a61f550c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:26.801877 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7095e93-241f-4db9-82a6-1f1a61f550c0 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:26.802159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7095e93-241f-4db9-82a6-1f1a61f550c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:26.802434 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7095e93-241f-4db9-82a6-1f1a61f550c0 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:26.803112 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7095e93-241f-4db9-82a6-1f1a61f550c0 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:26.804566 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7095e93-241f-4db9-82a6-1f1a61f550c0 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:26.808235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:26.828358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 498/984] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Oct 4 14:16:26 2022] POST /identity/v3/services => generated 242 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:16:26.906646 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b22c1b-e382-4028-a6e4-91dacda3c55e None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:26.961564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8297ffe9-b8fc-4ee2-86a1-40ec5310f06a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:26.962227 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8297ffe9-b8fc-4ee2-86a1-40ec5310f06a None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:26.962508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8297ffe9-b8fc-4ee2-86a1-40ec5310f06a None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:26.962783 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8297ffe9-b8fc-4ee2-86a1-40ec5310f06a None admin] PATH_INFO: `/v3/services/image` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:26.963392 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8297ffe9-b8fc-4ee2-86a1-40ec5310f06a None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:26.964905 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8297ffe9-b8fc-4ee2-86a1-40ec5310f06a None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:26.976120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-8297ffe9-b8fc-4ee2-86a1-40ec5310f06a None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 14:16:26.976120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:26.976120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:16:26.976120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:16:26.976120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:16:26.976120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:16:26.976120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:16:26.976120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:26.976120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 14:16:26.976120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:26.976120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:26.976120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:26.976120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:26.976120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:26.976120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:26.976120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:26.976120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:26.978263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:26.978263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:26.978263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:26.978263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:26.978263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:26.978263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:26.978263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:16:26.978263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:16:26.978263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:16:26.978263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:16:26.978263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:26.978263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:26.978263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:26.978263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:26.978263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:26.978263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:26.979919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:26.979919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:26.979919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:26.979919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:26.979919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:26.979919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:26.979919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:26.979919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:26.979919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:26.979919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:26.979919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:16:26.979919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:26.979919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 14:16:26.979919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:26.987371 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 487/985] 10.222.0.64 () {56 vars in 1105 bytes} [Tue Oct 4 14:16:26 2022] GET /identity/v3/services/image => generated 86 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:26.987649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7095e93-241f-4db9-82a6-1f1a61f550c0 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:27.165329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7095e93-241f-4db9-82a6-1f1a61f550c0 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:27.232867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd17a8f1-4e25-4fe1-b2ed-e23d23e2329f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:27.234361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd17a8f1-4e25-4fe1-b2ed-e23d23e2329f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:27.235129 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd17a8f1-4e25-4fe1-b2ed-e23d23e2329f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:27.235905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd17a8f1-4e25-4fe1-b2ed-e23d23e2329f None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:27.237394 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd17a8f1-4e25-4fe1-b2ed-e23d23e2329f None admin] RBAC: Adding query filter params (name=image) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:27.238238 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd17a8f1-4e25-4fe1-b2ed-e23d23e2329f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:27.241162 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd17a8f1-4e25-4fe1-b2ed-e23d23e2329f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:27.252237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 499/986] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:26 2022] GET /identity/v3/services?name=image => generated 122 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:27.261168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8297ffe9-b8fc-4ee2-86a1-40ec5310f06a None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:27.284837 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:27.479083 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8297ffe9-b8fc-4ee2-86a1-40ec5310f06a None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:27.558140 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0eb2a46-66e3-48f3-9bd9-eaad4dfae64a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:27.560844 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0eb2a46-66e3-48f3-9bd9-eaad4dfae64a None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:27.561317 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0eb2a46-66e3-48f3-9bd9-eaad4dfae64a None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:27.561794 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0eb2a46-66e3-48f3-9bd9-eaad4dfae64a None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:27.562999 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0eb2a46-66e3-48f3-9bd9-eaad4dfae64a None admin] RBAC: Adding query filter params (type=image) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:27.563538 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0eb2a46-66e3-48f3-9bd9-eaad4dfae64a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:27.567002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0eb2a46-66e3-48f3-9bd9-eaad4dfae64a None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:27.577957 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 488/987] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:27 2022] GET /identity/v3/services?type=image => generated 122 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:28.417215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18701d0-2d03-46f9-ab51-efe908059725 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:28.417757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18701d0-2d03-46f9-ab51-efe908059725 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:28.417847 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18701d0-2d03-46f9-ab51-efe908059725 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:28.419682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 500/988] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:28.425996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0898d10a-5617-44c6-b716-63a505ae4ba7 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:28.426513 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0898d10a-5617-44c6-b716-63a505ae4ba7 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:28.426590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0898d10a-5617-44c6-b716-63a505ae4ba7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:28.459268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-0898d10a-5617-44c6-b716-63a505ae4ba7 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:28.461372 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0898d10a-5617-44c6-b716-63a505ae4ba7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:28.573829 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 489/989] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:28 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:28.858175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18701d0-2d03-46f9-ab51-efe908059725 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:28.935099 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:28.994382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18701d0-2d03-46f9-ab51-efe908059725 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:29.038379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2e3a71-0c1f-4070-886f-1a0bda48ab3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:29.039031 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2e3a71-0c1f-4070-886f-1a0bda48ab3c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:29.039300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2e3a71-0c1f-4070-886f-1a0bda48ab3c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:29.039572 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2e3a71-0c1f-4070-886f-1a0bda48ab3c None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:29.040166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2e3a71-0c1f-4070-886f-1a0bda48ab3c None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:29.041582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2e3a71-0c1f-4070-886f-1a0bda48ab3c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:29.052357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-9b2e3a71-0c1f-4070-886f-1a0bda48ab3c None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 14:16:29.052357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:29.052357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:16:29.052357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:16:29.052357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:16:29.052357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:16:29.052357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:16:29.052357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:29.052357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 14:16:29.052357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:29.052357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:29.052357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:29.052357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:29.052357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:29.052357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:29.052357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:29.052357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:29.057020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:29.057020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:29.057020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:29.057020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:29.057020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:29.057020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:29.057020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:16:29.057020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:16:29.057020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:16:29.057020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:16:29.057020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:29.057020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:29.057020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:29.057020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:29.057020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:29.057020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:29.058082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:29.058082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:29.058082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:29.058082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:29.058082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:29.058082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:29.058082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:29.058082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:29.058082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:29.058082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:29.058082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:16:29.058082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:29.058082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 14:16:29.058082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:29.058082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 501/990] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Oct 4 14:16:28 2022] GET /identity/v3/services/object-store => generated 93 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:29.064761 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0898d10a-5617-44c6-b716-63a505ae4ba7 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:29.068851 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:29.226842 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0c99f2-531e-49a9-9602-82255d404c12 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:29.227104 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0c99f2-531e-49a9-9602-82255d404c12 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:29.227395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0c99f2-531e-49a9-9602-82255d404c12 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:29.228975 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 502/991] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:29.236436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ed03e7-88f8-4f05-94c1-11de0a92dbe2 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:29.236934 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ed03e7-88f8-4f05-94c1-11de0a92dbe2 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:29.237417 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ed03e7-88f8-4f05-94c1-11de0a92dbe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:29.253395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0898d10a-5617-44c6-b716-63a505ae4ba7 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:29.269671 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-65ed03e7-88f8-4f05-94c1-11de0a92dbe2 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:29.271730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-65ed03e7-88f8-4f05-94c1-11de0a92dbe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:29.295671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f97fd76-8686-45ac-bc98-352da35cc859 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:29.296301 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97fd76-8686-45ac-bc98-352da35cc859 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:29.296612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97fd76-8686-45ac-bc98-352da35cc859 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:29.296888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97fd76-8686-45ac-bc98-352da35cc859 None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:29.297521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f97fd76-8686-45ac-bc98-352da35cc859 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:29.297835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f97fd76-8686-45ac-bc98-352da35cc859 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:29.299232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f97fd76-8686-45ac-bc98-352da35cc859 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:29.307809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 490/992] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Oct 4 14:16:29 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:29.314180 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f97fd76-8686-45ac-bc98-352da35cc859 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:29.367019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 503/993] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:29 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:16:29.400005 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:29.444163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f97fd76-8686-45ac-bc98-352da35cc859 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:29.489542 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045bbcbb-2989-4dfe-9892-23188c435447 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:29.490135 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045bbcbb-2989-4dfe-9892-23188c435447 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:29.490418 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045bbcbb-2989-4dfe-9892-23188c435447 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:29.490743 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045bbcbb-2989-4dfe-9892-23188c435447 None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:29.491384 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045bbcbb-2989-4dfe-9892-23188c435447 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:29.491689 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045bbcbb-2989-4dfe-9892-23188c435447 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:29.493136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045bbcbb-2989-4dfe-9892-23188c435447 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:29.498938 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 491/994] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Oct 4 14:16:29 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:29.505328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ed03e7-88f8-4f05-94c1-11de0a92dbe2 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:29.570881 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:29.639219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ed03e7-88f8-4f05-94c1-11de0a92dbe2 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:29.709091 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e720d9-a5e8-4bef-b8c9-0dd2250e1944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:29.710270 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e720d9-a5e8-4bef-b8c9-0dd2250e1944 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:29.710850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e720d9-a5e8-4bef-b8c9-0dd2250e1944 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:29.711441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e720d9-a5e8-4bef-b8c9-0dd2250e1944 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:29.712920 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e720d9-a5e8-4bef-b8c9-0dd2250e1944 None admin] RBAC: Adding query filter params (interface=admin, service_id=5e7334324fd54270ae6f097612b336a0, region_id=RegionOne) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:29.713586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e720d9-a5e8-4bef-b8c9-0dd2250e1944 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:29.716880 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e720d9-a5e8-4bef-b8c9-0dd2250e1944 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:29.732778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 504/995] 10.222.0.64 () {56 vars in 1249 bytes} [Tue Oct 4 14:16:29 2022] GET /identity/v3/endpoints?service_id=5e7334324fd54270ae6f097612b336a0&interface=admin®ion_id=RegionOne => generated 193 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:29.733039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045bbcbb-2989-4dfe-9892-23188c435447 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:29.738017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e720d9-a5e8-4bef-b8c9-0dd2250e1944 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:29.919928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045bbcbb-2989-4dfe-9892-23188c435447 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:29.925853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e720d9-a5e8-4bef-b8c9-0dd2250e1944 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:29.962940 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:29.971536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:29.982452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5864d1b7-e267-40b3-845f-c0f290eece78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:29.983628 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5864d1b7-e267-40b3-845f-c0f290eece78 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:29.984164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5864d1b7-e267-40b3-845f-c0f290eece78 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:29.984713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5864d1b7-e267-40b3-845f-c0f290eece78 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:29.985650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5864d1b7-e267-40b3-845f-c0f290eece78 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:29.987815 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5864d1b7-e267-40b3-845f-c0f290eece78 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:30.000664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5561da60-0d65-48ee-91c7-95397293608c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:30.001624 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5561da60-0d65-48ee-91c7-95397293608c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:30.002087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5561da60-0d65-48ee-91c7-95397293608c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:30.002542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5561da60-0d65-48ee-91c7-95397293608c None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:30.003771 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5561da60-0d65-48ee-91c7-95397293608c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:30.006096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-5864d1b7-e267-40b3-845f-c0f290eece78 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 14:16:30.006096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:30.006096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:16:30.006096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:16:30.006096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:16:30.006096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:16:30.006096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:16:30.006096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:30.006096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 14:16:30.006096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:30.006096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:30.006096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:30.006096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:30.006096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:30.006096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:30.006096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:30.006096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:30.008834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:30.008834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:30.008834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:30.008834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:30.008834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:30.008834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:30.008834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:16:30.008834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:16:30.008834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:16:30.008834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:16:30.008834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:30.008834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:30.008834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:30.008834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:30.008834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:30.008834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:30.011287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:30.011287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:30.011287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:30.011287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:30.011287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:30.011287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:30.011287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:30.011287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:30.011287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:30.011287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:30.011287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:16:30.011287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:30.011287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 14:16:30.011287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:30.013721 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5561da60-0d65-48ee-91c7-95397293608c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:30.016808 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 492/996] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:16:29 2022] GET /identity/v3/services/compute => generated 88 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:30.020912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5864d1b7-e267-40b3-845f-c0f290eece78 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:30.021952 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 505/997] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Oct 4 14:16:29 2022] GET /identity/v3/services => generated 1727 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:16:30.171164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5864d1b7-e267-40b3-845f-c0f290eece78 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:30.218065 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3ce33c-9d25-4d8e-805f-324b33244eea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:30.218878 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3ce33c-9d25-4d8e-805f-324b33244eea None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:30.219259 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3ce33c-9d25-4d8e-805f-324b33244eea None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:30.219658 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3ce33c-9d25-4d8e-805f-324b33244eea None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:30.220466 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3ce33c-9d25-4d8e-805f-324b33244eea None admin] RBAC: Adding query filter params (name=compute) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:30.220876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3ce33c-9d25-4d8e-805f-324b33244eea None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:30.222420 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3ce33c-9d25-4d8e-805f-324b33244eea None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:30.228669 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 493/998] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Oct 4 14:16:30 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:30.236260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5561da60-0d65-48ee-91c7-95397293608c None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:30.350442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1ca1f4-a4ba-44ba-a25d-d1f39dacd97d None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:30.350843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1ca1f4-a4ba-44ba-a25d-d1f39dacd97d None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:30.351214 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1ca1f4-a4ba-44ba-a25d-d1f39dacd97d None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:30.353238 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 494/999] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:30.362086 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4c773a-28f1-4be2-ae7e-9a32472931dd None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:30.362529 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4c773a-28f1-4be2-ae7e-9a32472931dd None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:30.362966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4c773a-28f1-4be2-ae7e-9a32472931dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:30.392007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-ef4c773a-28f1-4be2-ae7e-9a32472931dd None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:30.393603 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ef4c773a-28f1-4be2-ae7e-9a32472931dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:30.406858 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5561da60-0d65-48ee-91c7-95397293608c None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:30.473651 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:30.481310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6364414f-b6d9-4a26-99dc-d8884390259c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:30.481948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6364414f-b6d9-4a26-99dc-d8884390259c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:30.482226 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6364414f-b6d9-4a26-99dc-d8884390259c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:30.482496 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6364414f-b6d9-4a26-99dc-d8884390259c None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:30.483180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6364414f-b6d9-4a26-99dc-d8884390259c None admin] RBAC: Adding query filter params (type=compute) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:30.483483 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6364414f-b6d9-4a26-99dc-d8884390259c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:30.484933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6364414f-b6d9-4a26-99dc-d8884390259c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:30.491612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 506/1000] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Oct 4 14:16:30 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:30.500354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6364414f-b6d9-4a26-99dc-d8884390259c None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:30.500607 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 495/1001] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:30 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:30.573878 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:30.677034 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6364414f-b6d9-4a26-99dc-d8884390259c None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:30.742780 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98b976b-fbe8-4582-8133-1eaa7370a6d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:30.743714 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98b976b-fbe8-4582-8133-1eaa7370a6d6 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:30.744188 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98b976b-fbe8-4582-8133-1eaa7370a6d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:30.744701 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98b976b-fbe8-4582-8133-1eaa7370a6d6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:30.746003 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98b976b-fbe8-4582-8133-1eaa7370a6d6 None admin] RBAC: Adding query filter params (interface=public, service_id=f3233a5bb97740b4b4b89d95fc182423, region_id=RegionOne) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:30.747103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98b976b-fbe8-4582-8133-1eaa7370a6d6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:30.749676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98b976b-fbe8-4582-8133-1eaa7370a6d6 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:30.764425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 507/1002] 10.222.0.64 () {56 vars in 1251 bytes} [Tue Oct 4 14:16:30 2022] GET /identity/v3/endpoints?service_id=f3233a5bb97740b4b4b89d95fc182423&interface=public®ion_id=RegionOne => generated 194 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:30.767265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4c773a-28f1-4be2-ae7e-9a32472931dd None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:30.948302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4c773a-28f1-4be2-ae7e-9a32472931dd None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:30.972525 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98b976b-fbe8-4582-8133-1eaa7370a6d6 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:31.020407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713f70e0-34dc-409b-b38e-b002c647ce75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:31.021289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713f70e0-34dc-409b-b38e-b002c647ce75 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:31.021726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713f70e0-34dc-409b-b38e-b002c647ce75 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:31.022163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713f70e0-34dc-409b-b38e-b002c647ce75 None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:31.023112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713f70e0-34dc-409b-b38e-b002c647ce75 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:31.025480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713f70e0-34dc-409b-b38e-b002c647ce75 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:31.037662 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 496/1003] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Oct 4 14:16:30 2022] GET /identity/v3/services => generated 1727 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:16:31.138793 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98b976b-fbe8-4582-8133-1eaa7370a6d6 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:31.151260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:31.188010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ace008-0f38-47bc-8852-9f04947d62e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:31.188875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ace008-0f38-47bc-8852-9f04947d62e5 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:31.189278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ace008-0f38-47bc-8852-9f04947d62e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:31.189691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ace008-0f38-47bc-8852-9f04947d62e5 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:31.190521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ace008-0f38-47bc-8852-9f04947d62e5 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:31.192139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ace008-0f38-47bc-8852-9f04947d62e5 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:31.215576 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 508/1004] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Oct 4 14:16:30 2022] POST /identity/v3/services => generated 242 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:16:32.571037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b8b259-29ed-4b0f-a012-b28e71c6facd None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:32.571879 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b8b259-29ed-4b0f-a012-b28e71c6facd None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:32.571879 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b8b259-29ed-4b0f-a012-b28e71c6facd None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:32.576357 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 497/1005] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:32.581595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064cce52-e2ee-4450-a18c-481a28f44864 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:32.582422 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064cce52-e2ee-4450-a18c-481a28f44864 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:32.582422 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064cce52-e2ee-4450-a18c-481a28f44864 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:32.602558 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-064cce52-e2ee-4450-a18c-481a28f44864 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:32.603797 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-064cce52-e2ee-4450-a18c-481a28f44864 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:32.712476 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 509/1006] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:32 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:33.006864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b8b259-29ed-4b0f-a012-b28e71c6facd None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:33.108380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:33.201087 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b8b259-29ed-4b0f-a012-b28e71c6facd None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:33.266857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14057e1c-95b2-4420-9f19-fc0dd2fc59ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:33.267875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14057e1c-95b2-4420-9f19-fc0dd2fc59ff None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:33.268404 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14057e1c-95b2-4420-9f19-fc0dd2fc59ff None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:33.268904 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14057e1c-95b2-4420-9f19-fc0dd2fc59ff None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:33.269972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14057e1c-95b2-4420-9f19-fc0dd2fc59ff None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:33.272653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14057e1c-95b2-4420-9f19-fc0dd2fc59ff None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:33.291116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-14057e1c-95b2-4420-9f19-fc0dd2fc59ff None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 14:16:33.291116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:33.291116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:16:33.291116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:16:33.291116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:16:33.291116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:16:33.291116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:16:33.291116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:33.291116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 14:16:33.291116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:33.291116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:33.291116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:33.291116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:33.291116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:33.291116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:33.291116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:33.291116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:33.292199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:33.292199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:33.292199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:33.292199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:33.292199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:33.292199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:33.292199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:16:33.292199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:16:33.292199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:16:33.292199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:16:33.292199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:33.292199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:33.292199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:33.292199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:33.292199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:33.292199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:33.293613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:33.293613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:33.293613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:33.293613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:33.293613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:33.293613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:33.293613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:33.293613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:33.293613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:33.293613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:33.293613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:16:33.293613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:33.293613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 14:16:33.293613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:33.298346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 498/1007] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Oct 4 14:16:33 2022] GET /identity/v3/services/object-store => generated 93 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:33.301660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064cce52-e2ee-4450-a18c-481a28f44864 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:33.464803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:33.475541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064cce52-e2ee-4450-a18c-481a28f44864 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:33.535428 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5630a8-ac4f-408c-b685-6f4bb066676c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:33.536902 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5630a8-ac4f-408c-b685-6f4bb066676c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:33.537625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5630a8-ac4f-408c-b685-6f4bb066676c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:33.538345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5630a8-ac4f-408c-b685-6f4bb066676c None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:33.539730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5630a8-ac4f-408c-b685-6f4bb066676c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:33.540689 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5630a8-ac4f-408c-b685-6f4bb066676c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:33.544120 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5630a8-ac4f-408c-b685-6f4bb066676c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:33.554250 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 510/1008] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Oct 4 14:16:33 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:33.561045 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14057e1c-95b2-4420-9f19-fc0dd2fc59ff None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:33.755809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14057e1c-95b2-4420-9f19-fc0dd2fc59ff None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:33.788333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:33.825307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0a88cf-efbd-4fba-a50c-0f482a0dd31f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:33.827160 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0a88cf-efbd-4fba-a50c-0f482a0dd31f None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:33.827970 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0a88cf-efbd-4fba-a50c-0f482a0dd31f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:33.828785 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0a88cf-efbd-4fba-a50c-0f482a0dd31f None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:33.830192 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0a88cf-efbd-4fba-a50c-0f482a0dd31f None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:33.831040 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0a88cf-efbd-4fba-a50c-0f482a0dd31f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:33.833991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0a88cf-efbd-4fba-a50c-0f482a0dd31f None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:33.844414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 499/1009] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Oct 4 14:16:33 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:33.851072 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5630a8-ac4f-408c-b685-6f4bb066676c None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:34.049600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5630a8-ac4f-408c-b685-6f4bb066676c None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:34.080706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6af7fe-6ffd-4862-93b4-4695a3474f46 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:34.081206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6af7fe-6ffd-4862-93b4-4695a3474f46 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:34.081687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6af7fe-6ffd-4862-93b4-4695a3474f46 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:34.084171 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 500/1010] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:34.090057 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e36010-bce5-4331-b875-97434b34460c None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:34.090573 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e36010-bce5-4331-b875-97434b34460c None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:34.091022 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e36010-bce5-4331-b875-97434b34460c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:34.112220 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a521f9cf-7fb1-4814-9011-28ef76c8c4a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:34.113263 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a521f9cf-7fb1-4814-9011-28ef76c8c4a5 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:34.113781 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a521f9cf-7fb1-4814-9011-28ef76c8c4a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:34.114200 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a521f9cf-7fb1-4814-9011-28ef76c8c4a5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:34.114397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-75e36010-bce5-4331-b875-97434b34460c None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:34.115463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a521f9cf-7fb1-4814-9011-28ef76c8c4a5 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:34.116198 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-75e36010-bce5-4331-b875-97434b34460c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:34.117867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a521f9cf-7fb1-4814-9011-28ef76c8c4a5 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:34.143035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 511/1011] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:16:33 2022] POST /identity/v3/endpoints => generated 325 bytes in 293 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:16:34.220522 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 501/1012] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:34 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:16:34.726057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a521f9cf-7fb1-4814-9011-28ef76c8c4a5 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:34.766570 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:34.931386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a1e8c4-eca6-4430-a95a-505baa0fe3b1 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:34.932214 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a1e8c4-eca6-4430-a95a-505baa0fe3b1 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:34.932435 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a1e8c4-eca6-4430-a95a-505baa0fe3b1 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:34.934510 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 502/1013] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:34.938478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a521f9cf-7fb1-4814-9011-28ef76c8c4a5 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:34.942931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcff120-8160-43f4-8797-8b1a25dba6ec None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:34.943439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcff120-8160-43f4-8797-8b1a25dba6ec None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:34.943921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcff120-8160-43f4-8797-8b1a25dba6ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:34.974542 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-fdcff120-8160-43f4-8797-8b1a25dba6ec None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:34.976572 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-fdcff120-8160-43f4-8797-8b1a25dba6ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:34.987594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3c2117-3d26-4f4a-9d0d-36aa0b616837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:34.988485 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3c2117-3d26-4f4a-9d0d-36aa0b616837 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:34.988900 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3c2117-3d26-4f4a-9d0d-36aa0b616837 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:34.989294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3c2117-3d26-4f4a-9d0d-36aa0b616837 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:34.990021 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3c2117-3d26-4f4a-9d0d-36aa0b616837 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:34.992340 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3c2117-3d26-4f4a-9d0d-36aa0b616837 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:35.005607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-2f3c2117-3d26-4f4a-9d0d-36aa0b616837 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 14:16:35.005607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:35.005607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:16:35.005607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:16:35.005607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:16:35.005607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:16:35.005607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:16:35.005607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:35.005607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 14:16:35.005607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:35.005607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:35.005607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:35.005607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:35.005607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:35.005607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:35.005607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:35.005607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:35.007844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:35.007844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:35.007844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:35.007844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:35.007844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:35.007844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:35.007844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:16:35.007844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:16:35.007844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:16:35.007844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:16:35.007844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:35.007844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:35.007844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:35.007844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:35.007844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:35.007844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:35.010028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:35.010028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:35.010028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:35.010028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:35.010028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:35.010028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:35.010028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:35.010028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:35.010028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:35.010028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:35.010028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:16:35.010028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:35.010028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 14:16:35.010028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:35.010028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 512/1014] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:16:34 2022] GET /identity/v3/services/compute => generated 88 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:35.016019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3c2117-3d26-4f4a-9d0d-36aa0b616837 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:35.111537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 503/1015] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:34 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:16:35.130257 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:35.158754 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3c2117-3d26-4f4a-9d0d-36aa0b616837 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:35.203488 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92e2307-6fc7-47be-97b3-0ba154fa7f7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:35.204102 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e2307-6fc7-47be-97b3-0ba154fa7f7a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:35.204411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e2307-6fc7-47be-97b3-0ba154fa7f7a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:35.204719 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e2307-6fc7-47be-97b3-0ba154fa7f7a None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:35.205338 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92e2307-6fc7-47be-97b3-0ba154fa7f7a None admin] RBAC: Adding query filter params (name=compute) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:35.205642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92e2307-6fc7-47be-97b3-0ba154fa7f7a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:35.207102 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92e2307-6fc7-47be-97b3-0ba154fa7f7a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:35.213936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 513/1016] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Oct 4 14:16:35 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:35.220587 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcff120-8160-43f4-8797-8b1a25dba6ec None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:35.265841 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:35.352993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcff120-8160-43f4-8797-8b1a25dba6ec None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:35.401745 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e2d37f-8583-4321-8e5f-9ee3069505b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:35.402498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e2d37f-8583-4321-8e5f-9ee3069505b4 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:35.402782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e2d37f-8583-4321-8e5f-9ee3069505b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:35.403059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e2d37f-8583-4321-8e5f-9ee3069505b4 None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:35.404026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e2d37f-8583-4321-8e5f-9ee3069505b4 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:35.404361 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e2d37f-8583-4321-8e5f-9ee3069505b4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:35.405804 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e2d37f-8583-4321-8e5f-9ee3069505b4 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:35.412137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 504/1017] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Oct 4 14:16:35 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:35.418864 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92e2307-6fc7-47be-97b3-0ba154fa7f7a None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:35.552180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92e2307-6fc7-47be-97b3-0ba154fa7f7a None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:35.585622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e2d37f-8583-4321-8e5f-9ee3069505b4 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:35.594773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508732d3-a65b-4dbc-af81-be56f171ddf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:35.595393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508732d3-a65b-4dbc-af81-be56f171ddf1 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:35.595667 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508732d3-a65b-4dbc-af81-be56f171ddf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:35.595936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508732d3-a65b-4dbc-af81-be56f171ddf1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:35.596671 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508732d3-a65b-4dbc-af81-be56f171ddf1 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:35.598096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508732d3-a65b-4dbc-af81-be56f171ddf1 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:35.629174 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 514/1018] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:16:35 2022] POST /identity/v3/endpoints => generated 334 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:16:35.786990 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:35.795809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e2d37f-8583-4321-8e5f-9ee3069505b4 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:35.869392 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f91b521-bfbc-4eae-8615-0974bd6828ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:35.870476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f91b521-bfbc-4eae-8615-0974bd6828ef None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:35.870969 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f91b521-bfbc-4eae-8615-0974bd6828ef None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:35.871441 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f91b521-bfbc-4eae-8615-0974bd6828ef None admin] PATH_INFO: `/v3/services/image` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:35.872488 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f91b521-bfbc-4eae-8615-0974bd6828ef None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:35.875073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f91b521-bfbc-4eae-8615-0974bd6828ef None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:35.893927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-4f91b521-bfbc-4eae-8615-0974bd6828ef None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 14:16:35.893927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:35.893927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:16:35.893927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:16:35.893927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:16:35.893927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:16:35.893927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:16:35.893927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:35.893927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 14:16:35.893927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:35.893927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:35.893927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:35.893927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:35.893927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:35.893927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:35.893927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:35.893927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:35.895015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:35.895015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:35.895015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:35.895015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:35.895015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:35.895015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:35.895015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:16:35.895015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:16:35.895015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:16:35.895015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:16:35.895015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:35.895015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:35.895015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:35.895015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:35.895015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:35.895015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:35.895875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:35.895875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:35.895875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:35.895875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:35.895875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:35.895875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:35.895875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:35.895875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:35.895875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:35.895875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:35.895875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:16:35.895875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:35.895875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 14:16:35.895875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:35.897322 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 505/1019] 10.222.0.64 () {56 vars in 1105 bytes} [Tue Oct 4 14:16:35 2022] GET /identity/v3/services/image => generated 86 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:35.906711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508732d3-a65b-4dbc-af81-be56f171ddf1 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:35.912231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:36.108111 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508732d3-a65b-4dbc-af81-be56f171ddf1 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:36.174346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd03b98b-9269-43fb-bcf2-f842a2f78ec7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:36.175112 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd03b98b-9269-43fb-bcf2-f842a2f78ec7 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:36.175581 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd03b98b-9269-43fb-bcf2-f842a2f78ec7 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:36.176053 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd03b98b-9269-43fb-bcf2-f842a2f78ec7 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:36.177247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd03b98b-9269-43fb-bcf2-f842a2f78ec7 None admin] RBAC: Adding query filter params (name=image) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:36.177766 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd03b98b-9269-43fb-bcf2-f842a2f78ec7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:36.180377 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd03b98b-9269-43fb-bcf2-f842a2f78ec7 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:36.190390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 515/1020] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:35 2022] GET /identity/v3/services?name=image => generated 122 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:36.196440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f91b521-bfbc-4eae-8615-0974bd6828ef None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:36.389603 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f91b521-bfbc-4eae-8615-0974bd6828ef None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:36.433335 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68cb9e7-8cd6-452c-afee-ef229413421e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:36.434505 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68cb9e7-8cd6-452c-afee-ef229413421e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:36.434980 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68cb9e7-8cd6-452c-afee-ef229413421e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:36.435262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68cb9e7-8cd6-452c-afee-ef229413421e None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:36.435920 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68cb9e7-8cd6-452c-afee-ef229413421e None admin] RBAC: Adding query filter params (type=image) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:36.436266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68cb9e7-8cd6-452c-afee-ef229413421e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:36.437714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68cb9e7-8cd6-452c-afee-ef229413421e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:36.444221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 506/1021] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:36 2022] GET /identity/v3/services?type=image => generated 350 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:36.452372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd03b98b-9269-43fb-bcf2-f842a2f78ec7 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:36.591862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:36.625446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f997e55e-df5e-47f7-8a1c-ce3150412b18 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:36.626242 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f997e55e-df5e-47f7-8a1c-ce3150412b18 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:36.626846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f997e55e-df5e-47f7-8a1c-ce3150412b18 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:36.629554 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 507/1022] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:36 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:36.638568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd615679-5066-4391-8290-4f5802e6145e None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:36.639034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd615679-5066-4391-8290-4f5802e6145e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:36.639565 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd615679-5066-4391-8290-4f5802e6145e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:36.662263 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd03b98b-9269-43fb-bcf2-f842a2f78ec7 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:36.672191 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-dd615679-5066-4391-8290-4f5802e6145e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:36.674270 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-dd615679-5066-4391-8290-4f5802e6145e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:36.705097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:36.733115 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846b70ed-7817-4503-8f52-deab5d0f2591 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:36.734160 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846b70ed-7817-4503-8f52-deab5d0f2591 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:36.734649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846b70ed-7817-4503-8f52-deab5d0f2591 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:36.735136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846b70ed-7817-4503-8f52-deab5d0f2591 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:36.736493 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846b70ed-7817-4503-8f52-deab5d0f2591 None admin] RBAC: Adding query filter params (interface=public, service_id=3fac5b9fcade4206995b426c2e217cb1, region_id=RegionOne) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:36.737029 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846b70ed-7817-4503-8f52-deab5d0f2591 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:36.740481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846b70ed-7817-4503-8f52-deab5d0f2591 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:36.760798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 516/1023] 10.222.0.64 () {56 vars in 1251 bytes} [Tue Oct 4 14:16:36 2022] GET /identity/v3/endpoints?service_id=3fac5b9fcade4206995b426c2e217cb1&interface=public®ion_id=RegionOne => generated 194 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:36.762975 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846b70ed-7817-4503-8f52-deab5d0f2591 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:36.773256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 508/1024] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:36 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:36.983314 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846b70ed-7817-4503-8f52-deab5d0f2591 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:37.045880 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:37.059430 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd615679-5066-4391-8290-4f5802e6145e None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:37.059987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961c1612-3bea-4f69-9a5d-078fe8ade02f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:37.061058 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961c1612-3bea-4f69-9a5d-078fe8ade02f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:37.061530 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961c1612-3bea-4f69-9a5d-078fe8ade02f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:37.061966 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961c1612-3bea-4f69-9a5d-078fe8ade02f None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:37.063060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961c1612-3bea-4f69-9a5d-078fe8ade02f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:37.065614 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961c1612-3bea-4f69-9a5d-078fe8ade02f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:37.088358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 517/1025] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Oct 4 14:16:36 2022] GET /identity/v3/services => generated 1957 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:16:37.201808 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd615679-5066-4391-8290-4f5802e6145e None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:37.247663 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123c1535-8a7e-463e-bdab-eae311513df5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:37.248212 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123c1535-8a7e-463e-bdab-eae311513df5 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:37.248541 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123c1535-8a7e-463e-bdab-eae311513df5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:37.248973 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123c1535-8a7e-463e-bdab-eae311513df5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:37.250060 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123c1535-8a7e-463e-bdab-eae311513df5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:37.251568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123c1535-8a7e-463e-bdab-eae311513df5 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:37.253609 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 509/1026] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:37 2022] GET /identity/v3/domains/default => generated 199 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:37.260173 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961c1612-3bea-4f69-9a5d-078fe8ade02f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:37.408798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961c1612-3bea-4f69-9a5d-078fe8ade02f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:37.489663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb7b4b2-a77b-4431-a9fb-a94f4f5b3778 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:37.495273 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb7b4b2-a77b-4431-a9fb-a94f4f5b3778 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:37.495919 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb7b4b2-a77b-4431-a9fb-a94f4f5b3778 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:37.496437 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb7b4b2-a77b-4431-a9fb-a94f4f5b3778 None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:37.497793 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb7b4b2-a77b-4431-a9fb-a94f4f5b3778 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:37.500626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb7b4b2-a77b-4431-a9fb-a94f4f5b3778 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:37.561573 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 518/1027] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Oct 4 14:16:37 2022] POST /identity/v3/projects => generated 311 bytes in 302 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:16:38.371542 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f87236a-ecb1-4639-b290-179472321edb None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:38.372380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f87236a-ecb1-4639-b290-179472321edb None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:38.372639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f87236a-ecb1-4639-b290-179472321edb None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:38.375091 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 510/1028] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:38 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:38.384176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133561c9-f48b-40a1-b424-ad99343c71d8 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:38.384176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133561c9-f48b-40a1-b424-ad99343c71d8 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:38.384176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133561c9-f48b-40a1-b424-ad99343c71d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:38.403489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-133561c9-f48b-40a1-b424-ad99343c71d8 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:38.404777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-133561c9-f48b-40a1-b424-ad99343c71d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:38.463756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:38.498316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 519/1029] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:38 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:16:38.779169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f87236a-ecb1-4639-b290-179472321edb None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:38.812835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:38.922191 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f87236a-ecb1-4639-b290-179472321edb None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:38.966722 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158ed6c7-1bbd-4e4e-9fac-d10cd979d221 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:38.967620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158ed6c7-1bbd-4e4e-9fac-d10cd979d221 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:38.968101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158ed6c7-1bbd-4e4e-9fac-d10cd979d221 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:38.968575 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158ed6c7-1bbd-4e4e-9fac-d10cd979d221 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:38.983420 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158ed6c7-1bbd-4e4e-9fac-d10cd979d221 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:16:38.988678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-158ed6c7-1bbd-4e4e-9fac-d10cd979d221 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 14:16:38.988678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:38.988678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:38.988678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:38.988678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:38.988678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:38.988678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:38.988678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:38.988678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:38.988678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:38.988678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:38.988678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:38.988678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:16:38.988678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:16:38.988678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:16:38.988678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:16:38.990406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:16:38.990406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:16:38.990406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:16:38.990406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:16:38.990406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:38.990406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:38.990406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:38.990406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:38.990406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:38.990406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:38.990406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:38.990406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:38.990406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:38.990406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:38.990406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:38.990406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:38.991695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:38.991695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:38.991695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:38.991695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:38.991695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:16:38.991695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:16:38.991695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:16:38.991695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:16:38.991695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:16:38.991695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:16:38.991695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 14:16:38.991695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:38.991695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 511/1030] 10.222.0.64 () {56 vars in 1120 bytes} [Tue Oct 4 14:16:38 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:38.996709 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133561c9-f48b-40a1-b424-ad99343c71d8 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:39.191918 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133561c9-f48b-40a1-b424-ad99343c71d8 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:39.259269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b016a7cc-9e95-45a3-a314-76ad428a1fc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:39.260324 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b016a7cc-9e95-45a3-a314-76ad428a1fc8 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:39.260829 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b016a7cc-9e95-45a3-a314-76ad428a1fc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:39.261301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b016a7cc-9e95-45a3-a314-76ad428a1fc8 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:39.263099 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b016a7cc-9e95-45a3-a314-76ad428a1fc8 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:39.263634 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b016a7cc-9e95-45a3-a314-76ad428a1fc8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:39.266139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b016a7cc-9e95-45a3-a314-76ad428a1fc8 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:39.286723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 520/1031] 10.222.0.64 () {56 vars in 1115 bytes} [Tue Oct 4 14:16:38 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:39.288464 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158ed6c7-1bbd-4e4e-9fac-d10cd979d221 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:39.400673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:39.424178 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158ed6c7-1bbd-4e4e-9fac-d10cd979d221 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:39.468228 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05aade47-b65c-439e-8857-c1d78477ef34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:39.468897 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aade47-b65c-439e-8857-c1d78477ef34 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:39.469181 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aade47-b65c-439e-8857-c1d78477ef34 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:39.469454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aade47-b65c-439e-8857-c1d78477ef34 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:39.482032 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05aade47-b65c-439e-8857-c1d78477ef34 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:39.483523 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05aade47-b65c-439e-8857-c1d78477ef34 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:39.500843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-05aade47-b65c-439e-8857-c1d78477ef34 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:39.500843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:39.500843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:39.500843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:39.500843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:39.500843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:39.500843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:39.500843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:39.500843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:39.500843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:39.500843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:39.500843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:39.500843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:39.500843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:39.500843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:39.500843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:39.501898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:39.501898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:39.501898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:39.501898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:39.501898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:39.501898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:39.501898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:39.501898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:39.501898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:39.501898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:39.501898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:39.501898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:39.501898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:39.501898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:39.501898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:39.501898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:39.501898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:39.502820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:39.502820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:39.502820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:39.502820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 512/1032] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:39.511820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b016a7cc-9e95-45a3-a314-76ad428a1fc8 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:39.595887 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:39.711042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b016a7cc-9e95-45a3-a314-76ad428a1fc8 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:39.775085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d57c90-f016-429e-98e7-1365b50d4c1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:39.776081 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d57c90-f016-429e-98e7-1365b50d4c1a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:39.776570 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d57c90-f016-429e-98e7-1365b50d4c1a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:39.777107 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d57c90-f016-429e-98e7-1365b50d4c1a None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:39.778532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d57c90-f016-429e-98e7-1365b50d4c1a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:39.779061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d57c90-f016-429e-98e7-1365b50d4c1a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:39.781571 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d57c90-f016-429e-98e7-1365b50d4c1a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:39.802794 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 521/1033] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:39.809593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05aade47-b65c-439e-8857-c1d78477ef34 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:39.828692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df49410-4b77-4422-93ed-55702962e678 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:39.829063 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df49410-4b77-4422-93ed-55702962e678 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:39.829251 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df49410-4b77-4422-93ed-55702962e678 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:39.833404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 522/1034] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:16:39.837663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10268dce-682f-4145-9f85-af37942d0023 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:39.838082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10268dce-682f-4145-9f85-af37942d0023 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:39.838545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10268dce-682f-4145-9f85-af37942d0023 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:39.860124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-10268dce-682f-4145-9f85-af37942d0023 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:39.861361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-10268dce-682f-4145-9f85-af37942d0023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:39.936510 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05aade47-b65c-439e-8857-c1d78477ef34 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:39.958779 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 523/1035] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:39 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:16:39.976921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730e0e71-8a8f-4f01-924d-20ac2d4c4a29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:39.977454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730e0e71-8a8f-4f01-924d-20ac2d4c4a29 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:39.977728 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730e0e71-8a8f-4f01-924d-20ac2d4c4a29 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:39.977998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730e0e71-8a8f-4f01-924d-20ac2d4c4a29 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:39.980558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:39.987265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730e0e71-8a8f-4f01-924d-20ac2d4c4a29 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:39.989675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730e0e71-8a8f-4f01-924d-20ac2d4c4a29 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:40.004168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-730e0e71-8a8f-4f01-924d-20ac2d4c4a29 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:16:40.004168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:40.004168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:40.004168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:40.004168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:40.004168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:40.004168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:40.004168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:40.004168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:40.004168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:40.004168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:40.004168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:40.004168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:16:40.004168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:16:40.004168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:16:40.004168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:40.013472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:40.015355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:40.015355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:40.015355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:16:40.015355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:16:40.015355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:16:40.015355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:16:40.015355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:16:40.015355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:16:40.015355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:16:40.015355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:40.015355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 513/1036] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Oct 4 14:16:39 2022] GET /identity/v3/users/nova => generated 82 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:40.020328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10268dce-682f-4145-9f85-af37942d0023 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:40.067453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:40.165228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10268dce-682f-4145-9f85-af37942d0023 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:40.174730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399de74d-9006-48da-9374-06da9550476c None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:40.175065 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399de74d-9006-48da-9374-06da9550476c None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:40.175331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399de74d-9006-48da-9374-06da9550476c None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:40.176717 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 514/1037] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:16:40.186494 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe3edf5-1ab9-41e3-ae28-d35b75cef19b None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:40.187033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe3edf5-1ab9-41e3-ae28-d35b75cef19b None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:40.187586 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe3edf5-1ab9-41e3-ae28-d35b75cef19b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:40.207494 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e82e183-e9c4-4abe-8407-8097d9c2b3d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:40.208774 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e82e183-e9c4-4abe-8407-8097d9c2b3d0 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:40.209058 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e82e183-e9c4-4abe-8407-8097d9c2b3d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:40.209329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e82e183-e9c4-4abe-8407-8097d9c2b3d0 None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:40.210082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e82e183-e9c4-4abe-8407-8097d9c2b3d0 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:40.210562 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e82e183-e9c4-4abe-8407-8097d9c2b3d0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:40.212018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e82e183-e9c4-4abe-8407-8097d9c2b3d0 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:40.216227 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-efe3edf5-1ab9-41e3-ae28-d35b75cef19b None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:40.218113 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-efe3edf5-1ab9-41e3-ae28-d35b75cef19b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:40.238823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 524/1038] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Oct 4 14:16:40 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:40.240946 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e82e183-e9c4-4abe-8407-8097d9c2b3d0 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:40.327356 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 515/1039] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:40 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:40.336291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe3edf5-1ab9-41e3-ae28-d35b75cef19b None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:40.356248 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:40.378795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e82e183-e9c4-4abe-8407-8097d9c2b3d0 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:40.420977 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918e0041-8a2e-4faa-aee0-21c855ee03c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:40.421601 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918e0041-8a2e-4faa-aee0-21c855ee03c1 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:40.421878 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918e0041-8a2e-4faa-aee0-21c855ee03c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:40.422148 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918e0041-8a2e-4faa-aee0-21c855ee03c1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:40.502880 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe3edf5-1ab9-41e3-ae28-d35b75cef19b None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:40.530809 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918e0041-8a2e-4faa-aee0-21c855ee03c1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:40.532345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918e0041-8a2e-4faa-aee0-21c855ee03c1 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:40.538973 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:40.556294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba843ca-eaba-4334-be6d-dbadfcd220b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:40.557170 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba843ca-eaba-4334-be6d-dbadfcd220b5 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:40.557579 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba843ca-eaba-4334-be6d-dbadfcd220b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:40.559603 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba843ca-eaba-4334-be6d-dbadfcd220b5 None admin] PATH_INFO: `/v3/services/image` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:40.560564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba843ca-eaba-4334-be6d-dbadfcd220b5 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:40.562213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba843ca-eaba-4334-be6d-dbadfcd220b5 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:40.574956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-eba843ca-eaba-4334-be6d-dbadfcd220b5 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 14:16:40.574956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:40.574956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:16:40.574956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:16:40.574956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:16:40.574956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:16:40.574956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:16:40.574956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:40.574956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 14:16:40.574956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:40.574956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:16:40.574956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:40.574956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:40.574956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:40.574956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:40.574956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:40.574956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:40.577363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:40.577363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:40.577363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:40.577363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:40.577363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:40.577363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:40.577363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:16:40.577363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:16:40.577363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:16:40.577363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:16:40.577363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:40.577363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:40.577363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:40.577363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:40.577363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:40.577363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:40.578407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:40.578407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:40.578407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:40.578407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:40.578407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:40.578407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:40.578407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:40.578407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:40.578407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:40.578407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:40.578407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:16:40.578407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:16:40.578407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 14:16:40.578407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:40.578407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 516/1040] 10.222.0.64 () {56 vars in 1105 bytes} [Tue Oct 4 14:16:40 2022] GET /identity/v3/services/image => generated 86 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:40.583134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba843ca-eaba-4334-be6d-dbadfcd220b5 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:40.647126 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:40.666377 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 525/1041] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:16:40 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 04 14:16:40.667244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918e0041-8a2e-4faa-aee0-21c855ee03c1 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:40.701327 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba843ca-eaba-4334-be6d-dbadfcd220b5 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:40.740536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79267de-ab4d-4536-9aa6-3fb1850e456e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:40.741168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79267de-ab4d-4536-9aa6-3fb1850e456e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:40.741453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79267de-ab4d-4536-9aa6-3fb1850e456e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:40.741735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79267de-ab4d-4536-9aa6-3fb1850e456e None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:40.742378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79267de-ab4d-4536-9aa6-3fb1850e456e None admin] RBAC: Adding query filter params (name=image) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:40.742683 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79267de-ab4d-4536-9aa6-3fb1850e456e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:40.744094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79267de-ab4d-4536-9aa6-3fb1850e456e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:40.756479 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 517/1042] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:40 2022] GET /identity/v3/services?name=image => generated 122 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:40.756479 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79267de-ab4d-4536-9aa6-3fb1850e456e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:40.783325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918e0041-8a2e-4faa-aee0-21c855ee03c1 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:40.814890 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:40.823173 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d480df13-09e8-4964-8e67-a43713e28e1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:40.823898 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d480df13-09e8-4964-8e67-a43713e28e1b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:40.824629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d480df13-09e8-4964-8e67-a43713e28e1b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:40.824629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d480df13-09e8-4964-8e67-a43713e28e1b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:40.825822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d480df13-09e8-4964-8e67-a43713e28e1b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:40.827289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d480df13-09e8-4964-8e67-a43713e28e1b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:40.829029 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 526/1043] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:40.829849 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d480df13-09e8-4964-8e67-a43713e28e1b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:40.877541 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79267de-ab4d-4536-9aa6-3fb1850e456e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:40.901805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:40.927791 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de474e9-611b-4cb7-88ad-2c109823a6e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:40.928613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de474e9-611b-4cb7-88ad-2c109823a6e6 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:40.928897 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de474e9-611b-4cb7-88ad-2c109823a6e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:40.929174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de474e9-611b-4cb7-88ad-2c109823a6e6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:40.941249 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de474e9-611b-4cb7-88ad-2c109823a6e6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:40.943019 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de474e9-611b-4cb7-88ad-2c109823a6e6 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:40.959900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-0de474e9-611b-4cb7-88ad-2c109823a6e6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:40.959900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:40.959900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:40.959900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:40.959900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:40.959900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:40.959900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:40.959900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:40.959900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:40.959900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:40.959900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:40.959900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:40.959900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:40.959900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:40.959900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:40.959900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:40.961864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:40.961864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:40.961864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:40.961864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:40.961864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:40.961864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:40.961864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:40.961864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:40.961864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:40.961864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:40.961864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:40.961864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:40.961864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:40.961864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:40.961864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:40.961864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:40.961864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:40.963658 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:40.963658 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:40.963658 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:40.964005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d480df13-09e8-4964-8e67-a43713e28e1b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:40.964143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 518/1044] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:40.964143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de474e9-611b-4cb7-88ad-2c109823a6e6 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:41.006125 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b02a11e-608c-442f-b6df-2f2f4c2c9688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:41.006891 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b02a11e-608c-442f-b6df-2f2f4c2c9688 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:41.007124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b02a11e-608c-442f-b6df-2f2f4c2c9688 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:41.007487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b02a11e-608c-442f-b6df-2f2f4c2c9688 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:41.008232 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b02a11e-608c-442f-b6df-2f2f4c2c9688 None admin] RBAC: Adding query filter params (type=image) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:41.008562 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b02a11e-608c-442f-b6df-2f2f4c2c9688 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:41.010204 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b02a11e-608c-442f-b6df-2f2f4c2c9688 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:41.016291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 527/1045] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:40 2022] GET /identity/v3/services?type=image => generated 350 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:41.017175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b02a11e-608c-442f-b6df-2f2f4c2c9688 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:41.102531 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de474e9-611b-4cb7-88ad-2c109823a6e6 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:41.106614 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:41.148814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8e2f74-06b0-488c-b8ed-63d98d133f4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:41.149424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8e2f74-06b0-488c-b8ed-63d98d133f4d None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:41.149704 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8e2f74-06b0-488c-b8ed-63d98d133f4d None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:41.149978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8e2f74-06b0-488c-b8ed-63d98d133f4d None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:41.150647 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8e2f74-06b0-488c-b8ed-63d98d133f4d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:41.152198 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8e2f74-06b0-488c-b8ed-63d98d133f4d None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:41.161308 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b02a11e-608c-442f-b6df-2f2f4c2c9688 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:41.189916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 519/1046] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:16:40 2022] POST /identity/v3/users => generated 285 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:16:41.189916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8e2f74-06b0-488c-b8ed-63d98d133f4d None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:41.201273 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:41.228201 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94621448-6def-46ea-9690-2bc812fb5a07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:41.229419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94621448-6def-46ea-9690-2bc812fb5a07 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:41.229901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94621448-6def-46ea-9690-2bc812fb5a07 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:41.230413 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94621448-6def-46ea-9690-2bc812fb5a07 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:41.231666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94621448-6def-46ea-9690-2bc812fb5a07 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:41.232221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94621448-6def-46ea-9690-2bc812fb5a07 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:41.234929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94621448-6def-46ea-9690-2bc812fb5a07 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:41.257963 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 528/1047] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:41.265577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94621448-6def-46ea-9690-2bc812fb5a07 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:41.334831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8e2f74-06b0-488c-b8ed-63d98d133f4d None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:41.380917 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3815bc-95e4-46fd-bbe6-7f3135b2b0ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:41.381548 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3815bc-95e4-46fd-bbe6-7f3135b2b0ce None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:41.381830 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3815bc-95e4-46fd-bbe6-7f3135b2b0ce None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:41.382197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3815bc-95e4-46fd-bbe6-7f3135b2b0ce None admin] PATH_INFO: `/v3/endpoints` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:41.382926 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3815bc-95e4-46fd-bbe6-7f3135b2b0ce None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:41.384442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3815bc-95e4-46fd-bbe6-7f3135b2b0ce None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:41.416369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 520/1048] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:16:41 2022] POST /identity/v3/endpoints => generated 327 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:16:41.426013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94621448-6def-46ea-9690-2bc812fb5a07 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:41.515849 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d852a92-7a34-40ce-bc87-3f4f50c286e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:41.516828 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d852a92-7a34-40ce-bc87-3f4f50c286e2 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:41.517290 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d852a92-7a34-40ce-bc87-3f4f50c286e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:41.517777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d852a92-7a34-40ce-bc87-3f4f50c286e2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:41.533107 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d852a92-7a34-40ce-bc87-3f4f50c286e2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:41.535921 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d852a92-7a34-40ce-bc87-3f4f50c286e2 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:41.560954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-8d852a92-7a34-40ce-bc87-3f4f50c286e2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:41.560954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:41.560954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:41.560954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:41.560954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:41.560954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:41.560954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:41.560954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:41.560954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:41.560954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:41.560954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:41.560954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:41.560954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:16:41.560954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:16:41.560954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:16:41.563007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:16:41.563007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:41.563007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:41.563007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:41.563007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:41.563007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:41.563007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:41.563007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:41.563007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:41.563007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:41.563007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:41.563007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:41.563007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:41.563007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:41.563007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:41.563007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:41.563007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:41.565241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:16:41.565241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:16:41.565241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:16:41.565241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:16:41.565241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:16:41.565241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:16:41.565241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:41.565241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:41.565241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 529/1049] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:16:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:41.571687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3815bc-95e4-46fd-bbe6-7f3135b2b0ce None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:41.598826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:41.762629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3815bc-95e4-46fd-bbe6-7f3135b2b0ce None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:41.827866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346163b0-4141-404c-8c02-2eef90426494 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:41.828853 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346163b0-4141-404c-8c02-2eef90426494 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:41.829350 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346163b0-4141-404c-8c02-2eef90426494 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:41.829824 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346163b0-4141-404c-8c02-2eef90426494 None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:41.831139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346163b0-4141-404c-8c02-2eef90426494 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:41.831662 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346163b0-4141-404c-8c02-2eef90426494 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:41.834594 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346163b0-4141-404c-8c02-2eef90426494 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:41.865023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 521/1050] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:16:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:41.867487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d852a92-7a34-40ce-bc87-3f4f50c286e2 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:41.929685 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:42.024973 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d852a92-7a34-40ce-bc87-3f4f50c286e2 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:42.073020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76832baf-713b-4730-b700-a4eb5da4029a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:42.073651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76832baf-713b-4730-b700-a4eb5da4029a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:42.074523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76832baf-713b-4730-b700-a4eb5da4029a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:42.074902 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76832baf-713b-4730-b700-a4eb5da4029a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:42.075999 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76832baf-713b-4730-b700-a4eb5da4029a None admin] RBAC: Adding query filter params (role.id=47e35bb0c3b44184986f4e28032e8c17, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=03081d7d581745c88c12e9d089fcb2d3) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:42.076380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76832baf-713b-4730-b700-a4eb5da4029a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:42.077841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76832baf-713b-4730-b700-a4eb5da4029a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:42.085149 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 530/1051] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:16:41 2022] GET /identity/v3/role_assignments?user.id=03081d7d581745c88c12e9d089fcb2d3&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=47e35bb0c3b44184986f4e28032e8c17 => generated 259 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:43.760299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064357ec-cd48-476b-94ed-63e81c6fcd37 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:43.760826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064357ec-cd48-476b-94ed-63e81c6fcd37 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:43.761857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064357ec-cd48-476b-94ed-63e81c6fcd37 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:43.764411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 522/1052] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:43 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:43.770503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a16f68-6f25-47eb-918e-b4e59dc6635d None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:43.771313 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a16f68-6f25-47eb-918e-b4e59dc6635d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:43.771313 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a16f68-6f25-47eb-918e-b4e59dc6635d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:43.793138 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-76a16f68-6f25-47eb-918e-b4e59dc6635d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:43.794367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-76a16f68-6f25-47eb-918e-b4e59dc6635d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:43.885505 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 531/1053] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:16:43.948453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801b1357-0a08-4099-a845-4e18d037abe2 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:43.948725 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801b1357-0a08-4099-a845-4e18d037abe2 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:43.949169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801b1357-0a08-4099-a845-4e18d037abe2 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:43.951158 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 523/1054] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:43.956936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc50d2d-1dff-4620-a123-77e0a1d4f68f None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:43.957311 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc50d2d-1dff-4620-a123-77e0a1d4f68f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:43.957478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc50d2d-1dff-4620-a123-77e0a1d4f68f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:43.978087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-fdc50d2d-1dff-4620-a123-77e0a1d4f68f None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:43.979318 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-fdc50d2d-1dff-4620-a123-77e0a1d4f68f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:44.015635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:44.069655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 532/1055] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:44.158475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801b1357-0a08-4099-a845-4e18d037abe2 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:44.252582 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:44.284438 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801b1357-0a08-4099-a845-4e18d037abe2 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:44.330015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef78064-16e7-4c63-930b-d2d940104d05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:44.330773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef78064-16e7-4c63-930b-d2d940104d05 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:44.331110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef78064-16e7-4c63-930b-d2d940104d05 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:44.331358 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef78064-16e7-4c63-930b-d2d940104d05 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:44.422006 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef78064-16e7-4c63-930b-d2d940104d05 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:44.423537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef78064-16e7-4c63-930b-d2d940104d05 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:44.523110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:44.544683 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc50d2d-1dff-4620-a123-77e0a1d4f68f None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:44.553225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 524/1056] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:16:44 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 14:16:44.559664 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef78064-16e7-4c63-930b-d2d940104d05 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:44.713112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef78064-16e7-4c63-930b-d2d940104d05 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:44.718095 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc50d2d-1dff-4620-a123-77e0a1d4f68f None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:44.764409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d22f28-a790-4b2d-9c2b-30eb8fca02d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:44.765248 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d22f28-a790-4b2d-9c2b-30eb8fca02d4 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:44.765409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d22f28-a790-4b2d-9c2b-30eb8fca02d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:44.765563 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d22f28-a790-4b2d-9c2b-30eb8fca02d4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:44.778907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0914fe7e-26f6-4a6d-b990-367d6ca1a854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:44.779752 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d22f28-a790-4b2d-9c2b-30eb8fca02d4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:44.779913 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0914fe7e-26f6-4a6d-b990-367d6ca1a854 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:44.780288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0914fe7e-26f6-4a6d-b990-367d6ca1a854 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:44.780573 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d22f28-a790-4b2d-9c2b-30eb8fca02d4 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:44.780803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0914fe7e-26f6-4a6d-b990-367d6ca1a854 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:44.797742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-70d22f28-a790-4b2d-9c2b-30eb8fca02d4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:44.797742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:44.797742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:44.797742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:44.797742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:44.797742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:44.797742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:44.797742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:44.797742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:44.797742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:44.797742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:44.797742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:44.797742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:44.797742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:44.797742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:44.797742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:44.799105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:44.799105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:44.799105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:44.799105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:44.799105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:44.799105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:44.799105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:44.799105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:44.799105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:44.799105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:44.799105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:44.799105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:44.799105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:44.799105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:44.799105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:44.799105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:44.799105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:44.800926 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:44.800926 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:44.800926 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:44.800926 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 525/1057] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:44.803329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0914fe7e-26f6-4a6d-b990-367d6ca1a854 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:16:44.805099 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d22f28-a790-4b2d-9c2b-30eb8fca02d4 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:44.813367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-0914fe7e-26f6-4a6d-b990-367d6ca1a854 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:16:44.813367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:44.813367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:44.813367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:44.813367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:44.813367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:44.813367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:44.813367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:44.813367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:44.813367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:44.813367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:44.813367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:44.813367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:16:44.813367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:16:44.813367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:16:44.813367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:16:44.817688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:16:44.817688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:16:44.817688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:16:44.817688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:16:44.817688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:44.817688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:44.817688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:44.817688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:44.817688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:44.817688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:44.817688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:44.817688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:44.817688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:44.817688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:44.817688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:44.817688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:44.820925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:44.820925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:44.820925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:44.820925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:44.820925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:16:44.820925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:16:44.820925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:16:44.820925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:16:44.820925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:16:44.820925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:16:44.820925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:16:44.820925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:44.824179 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 533/1058] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:16:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:44.825518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0914fe7e-26f6-4a6d-b990-367d6ca1a854 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:44.849749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:44.864588 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:44.943361 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d22f28-a790-4b2d-9c2b-30eb8fca02d4 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:44.978415 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0914fe7e-26f6-4a6d-b990-367d6ca1a854 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:45.001657 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d190249-3f3e-4441-8bc1-d007711b9d52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:45.002453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d190249-3f3e-4441-8bc1-d007711b9d52 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:45.002861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d190249-3f3e-4441-8bc1-d007711b9d52 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:45.003262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d190249-3f3e-4441-8bc1-d007711b9d52 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:45.004380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d190249-3f3e-4441-8bc1-d007711b9d52 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:45.004836 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d190249-3f3e-4441-8bc1-d007711b9d52 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:45.006871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d190249-3f3e-4441-8bc1-d007711b9d52 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:45.021108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a68bcb9-2de8-4fe5-80fb-829850289be3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:45.021734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a68bcb9-2de8-4fe5-80fb-829850289be3 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:45.022009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a68bcb9-2de8-4fe5-80fb-829850289be3 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:45.022279 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a68bcb9-2de8-4fe5-80fb-829850289be3 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:45.022961 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a68bcb9-2de8-4fe5-80fb-829850289be3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:45.023262 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a68bcb9-2de8-4fe5-80fb-829850289be3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:45.024016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 526/1059] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:45.029457 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a68bcb9-2de8-4fe5-80fb-829850289be3 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:45.041800 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 534/1060] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:16:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:45.049127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d190249-3f3e-4441-8bc1-d007711b9d52 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:45.061833 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f549826-3b6a-4c08-a8f6-ec91d37d676c None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:45.062101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f549826-3b6a-4c08-a8f6-ec91d37d676c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:45.062387 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f549826-3b6a-4c08-a8f6-ec91d37d676c None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:45.064129 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 535/1061] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:16:45.071755 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d869a48-a942-4c6f-a44f-03b67a3fb65c None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:45.072059 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d869a48-a942-4c6f-a44f-03b67a3fb65c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:45.072419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d869a48-a942-4c6f-a44f-03b67a3fb65c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:45.094602 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-9d869a48-a942-4c6f-a44f-03b67a3fb65c None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:45.095865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9d869a48-a942-4c6f-a44f-03b67a3fb65c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:45.175639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:45.192293 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d190249-3f3e-4441-8bc1-d007711b9d52 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:45.200380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 536/1062] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:16:45.236108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe3ae4a-d76b-490c-bd08-d64562bcf917 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:45.236796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe3ae4a-d76b-490c-bd08-d64562bcf917 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:45.237073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe3ae4a-d76b-490c-bd08-d64562bcf917 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:45.237336 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe3ae4a-d76b-490c-bd08-d64562bcf917 None admin] PATH_INFO: `/v3/users/ca859d18d0da4f85be4a6d6062e5ddb7` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:45.255064 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe3ae4a-d76b-490c-bd08-d64562bcf917 None admin] RBAC: Authorizing `identity:get_user(user_id=ca859d18d0da4f85be4a6d6062e5ddb7)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:45.257378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe3ae4a-d76b-490c-bd08-d64562bcf917 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:45.259395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 527/1063] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:16:45 2022] GET /identity/v3/users/ca859d18d0da4f85be4a6d6062e5ddb7 => generated 285 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:45.266542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d869a48-a942-4c6f-a44f-03b67a3fb65c None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:45.288195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:45.435310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d869a48-a942-4c6f-a44f-03b67a3fb65c None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:45.494616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe3ae4a-d76b-490c-bd08-d64562bcf917 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:45.494857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae2b12f-f268-4a05-a32e-6b8d389670e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:45.496417 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae2b12f-f268-4a05-a32e-6b8d389670e6 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:45.496872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae2b12f-f268-4a05-a32e-6b8d389670e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:45.497347 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae2b12f-f268-4a05-a32e-6b8d389670e6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:45.609955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae2b12f-f268-4a05-a32e-6b8d389670e6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:45.613171 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae2b12f-f268-4a05-a32e-6b8d389670e6 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:45.617165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe3ae4a-d76b-490c-bd08-d64562bcf917 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:45.629083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:45.672715 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d22a29-d20c-4b95-a139-433e4ea526a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:45.673651 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d22a29-d20c-4b95-a139-433e4ea526a5 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:45.674078 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d22a29-d20c-4b95-a139-433e4ea526a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:45.674495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d22a29-d20c-4b95-a139-433e4ea526a5 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:45.691049 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d22a29-d20c-4b95-a139-433e4ea526a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:16:45.700033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-b2d22a29-d20c-4b95-a139-433e4ea526a5 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 14:16:45.700033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:45.700033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:45.700033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:45.700033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:45.700033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:45.700033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:45.700033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:45.700033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:45.700033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:45.700033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:45.700033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:45.700033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:16:45.700033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:16:45.700033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:16:45.700033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:16:45.701148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:16:45.701148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:16:45.701148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:16:45.701148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:16:45.701148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:45.701148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:45.701148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:45.701148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:45.701148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:45.701148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:45.701148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:45.701148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:45.701148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:45.701148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:45.701148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:45.701148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:45.702017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:45.702017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:45.702017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:45.702017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:45.702017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:16:45.702017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:16:45.702017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:16:45.702017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:16:45.702017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:16:45.702017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:16:45.702017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 14:16:45.702017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:45.702017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 528/1064] 10.222.0.64 () {56 vars in 1120 bytes} [Tue Oct 4 14:16:45 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:45.707796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d22a29-d20c-4b95-a139-433e4ea526a5 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:45.717849 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:45.739558 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 537/1065] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:16:45 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 474 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 14:16:45.745933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae2b12f-f268-4a05-a32e-6b8d389670e6 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:45.832325 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d22a29-d20c-4b95-a139-433e4ea526a5 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:45.862649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae2b12f-f268-4a05-a32e-6b8d389670e6 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:45.873144 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9893231d-840c-47f7-b969-747d7fac854a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:45.873773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9893231d-840c-47f7-b969-747d7fac854a None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:45.874055 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9893231d-840c-47f7-b969-747d7fac854a None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:45.874328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9893231d-840c-47f7-b969-747d7fac854a None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:45.875003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9893231d-840c-47f7-b969-747d7fac854a None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:45.875304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9893231d-840c-47f7-b969-747d7fac854a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:45.876377 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:45.876900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9893231d-840c-47f7-b969-747d7fac854a None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:45.896421 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 529/1066] 10.222.0.64 () {56 vars in 1115 bytes} [Tue Oct 4 14:16:45 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:45.897975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9893231d-840c-47f7-b969-747d7fac854a None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:45.910320 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ec8e4c-39b0-42d9-9c61-45a8d0dc087e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:45.910963 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ec8e4c-39b0-42d9-9c61-45a8d0dc087e None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:45.911240 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ec8e4c-39b0-42d9-9c61-45a8d0dc087e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:45.911512 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ec8e4c-39b0-42d9-9c61-45a8d0dc087e None admin] PATH_INFO: `/v3/projects/585f5846d80a486cb51074489e643309` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:45.912748 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ec8e4c-39b0-42d9-9c61-45a8d0dc087e None admin] RBAC: Authorizing `identity:get_project(project_id=585f5846d80a486cb51074489e643309)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:45.914203 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ec8e4c-39b0-42d9-9c61-45a8d0dc087e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:45.916241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 538/1067] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:16:45 2022] GET /identity/v3/projects/585f5846d80a486cb51074489e643309 => generated 311 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:45.922651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ec8e4c-39b0-42d9-9c61-45a8d0dc087e None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:45.979299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:46.019298 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9893231d-840c-47f7-b969-747d7fac854a None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:46.040572 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ec8e4c-39b0-42d9-9c61-45a8d0dc087e None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:46.069687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbb1a67-984d-4abb-9271-e9c41dd4cecd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:46.070687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbb1a67-984d-4abb-9271-e9c41dd4cecd None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:46.071141 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbb1a67-984d-4abb-9271-e9c41dd4cecd None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:46.071591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbb1a67-984d-4abb-9271-e9c41dd4cecd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:46.089993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbb1a67-984d-4abb-9271-e9c41dd4cecd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:46.092857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbb1a67-984d-4abb-9271-e9c41dd4cecd None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:46.095409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6472538-1e7b-4c72-bfbf-b88e0edbead7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:46.096303 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6472538-1e7b-4c72-bfbf-b88e0edbead7 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:46.096799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6472538-1e7b-4c72-bfbf-b88e0edbead7 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:46.097206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6472538-1e7b-4c72-bfbf-b88e0edbead7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:46.098414 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6472538-1e7b-4c72-bfbf-b88e0edbead7 None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=585f5846d80a486cb51074489e643309, user.id=ca859d18d0da4f85be4a6d6062e5ddb7) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:46.098912 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6472538-1e7b-4c72-bfbf-b88e0edbead7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:46.101479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6472538-1e7b-4c72-bfbf-b88e0edbead7 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:46.111517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 539/1068] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:16:45 2022] GET /identity/v3/role_assignments?user.id=ca859d18d0da4f85be4a6d6062e5ddb7&scope.project.id=585f5846d80a486cb51074489e643309&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 259 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:46.116110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-6fbb1a67-984d-4abb-9271-e9c41dd4cecd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:46.116110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:46.116110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:46.116110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:46.116110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:46.116110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:46.116110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:46.116110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:46.116110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:46.116110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:46.116110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:46.116110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:46.116110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:46.116110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:46.116110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:46.116110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:46.118204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:46.118204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:46.118204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:46.118204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:46.118204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:46.118204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:46.118204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:46.118204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:46.118204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:46.118204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:46.118204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:46.118204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:46.118204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:46.118204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:46.118204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:46.118204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:46.118204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:46.120914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:46.120914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:46.120914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:46.120914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 530/1069] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:46.125735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6472538-1e7b-4c72-bfbf-b88e0edbead7 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:46.181649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:46.281243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6472538-1e7b-4c72-bfbf-b88e0edbead7 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:46.342471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7185403e-9430-434c-8cd2-682a45dec491 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:46.343426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7185403e-9430-434c-8cd2-682a45dec491 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:46.343865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7185403e-9430-434c-8cd2-682a45dec491 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:46.344348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7185403e-9430-434c-8cd2-682a45dec491 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:46.345429 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7185403e-9430-434c-8cd2-682a45dec491 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:46.345922 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7185403e-9430-434c-8cd2-682a45dec491 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:46.348249 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7185403e-9430-434c-8cd2-682a45dec491 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:46.365769 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 540/1070] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:46.372860 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbb1a67-984d-4abb-9271-e9c41dd4cecd None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:46.492444 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbb1a67-984d-4abb-9271-e9c41dd4cecd None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:46.516791 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:46.534375 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec557663-e02a-44bb-a36a-dcf0f34904a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:46.534935 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec557663-e02a-44bb-a36a-dcf0f34904a1 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:46.535221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec557663-e02a-44bb-a36a-dcf0f34904a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:46.535501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec557663-e02a-44bb-a36a-dcf0f34904a1 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:46.544446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec557663-e02a-44bb-a36a-dcf0f34904a1 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:46.545932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec557663-e02a-44bb-a36a-dcf0f34904a1 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:46.559465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-ec557663-e02a-44bb-a36a-dcf0f34904a1 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:16:46.559465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:46.559465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:46.559465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:46.559465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:46.559465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:46.559465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:46.559465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:46.559465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:46.559465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:46.559465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:46.559465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:46.559465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:16:46.559465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:16:46.559465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:16:46.559465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:46.561307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:46.563165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:46.563165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:46.563165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:16:46.563165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:16:46.563165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:16:46.563165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:16:46.563165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:16:46.563165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:16:46.563165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:16:46.563165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:46.563165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 531/1071] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Oct 4 14:16:46 2022] GET /identity/v3/users/nova => generated 82 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:46.568464 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7185403e-9430-434c-8cd2-682a45dec491 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:46.695601 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7185403e-9430-434c-8cd2-682a45dec491 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:46.697454 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:46.741744 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19a6923-601a-49b9-a3fe-a0e1a2a0ee74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:46.742435 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19a6923-601a-49b9-a3fe-a0e1a2a0ee74 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:46.742770 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19a6923-601a-49b9-a3fe-a0e1a2a0ee74 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:46.743109 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19a6923-601a-49b9-a3fe-a0e1a2a0ee74 None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:46.743921 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19a6923-601a-49b9-a3fe-a0e1a2a0ee74 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:46.744278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19a6923-601a-49b9-a3fe-a0e1a2a0ee74 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:46.746884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19a6923-601a-49b9-a3fe-a0e1a2a0ee74 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:46.769183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 541/1072] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Oct 4 14:16:46 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:46.775942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec557663-e02a-44bb-a36a-dcf0f34904a1 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:46.935769 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec557663-e02a-44bb-a36a-dcf0f34904a1 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:46.976011 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3624f424-c602-4bda-b32e-5d92f03a9829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:46.976825 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3624f424-c602-4bda-b32e-5d92f03a9829 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:46.977226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3624f424-c602-4bda-b32e-5d92f03a9829 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:46.977580 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3624f424-c602-4bda-b32e-5d92f03a9829 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:46.994753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3624f424-c602-4bda-b32e-5d92f03a9829 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:46.996609 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3624f424-c602-4bda-b32e-5d92f03a9829 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:47.014314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-3624f424-c602-4bda-b32e-5d92f03a9829 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:47.014314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:47.014314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:47.014314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:47.014314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:47.014314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:47.014314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:47.014314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:47.014314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:47.014314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:47.014314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:47.014314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:47.014314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:47.014314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:47.014314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:47.014314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:47.016434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:47.016434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:47.016434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:47.016434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:47.016434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:47.016434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:47.016434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:47.016434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:47.016434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:47.016434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:47.016434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:47.016434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:47.016434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:47.016434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:47.016434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:47.016434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:47.016434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:47.018269 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:47.018269 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:47.018269 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:47.018269 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 532/1073] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:47.030257 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19a6923-601a-49b9-a3fe-a0e1a2a0ee74 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:47.146805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19a6923-601a-49b9-a3fe-a0e1a2a0ee74 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:47.191305 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d36607-3856-4097-999b-22626a060120 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:47.191841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d36607-3856-4097-999b-22626a060120 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:47.192115 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d36607-3856-4097-999b-22626a060120 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:47.192418 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d36607-3856-4097-999b-22626a060120 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:47.193089 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d36607-3856-4097-999b-22626a060120 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:47.193389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d36607-3856-4097-999b-22626a060120 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:47.194811 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d36607-3856-4097-999b-22626a060120 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:47.209781 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 542/1074] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:47.216852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3624f424-c602-4bda-b32e-5d92f03a9829 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:47.276953 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:47.365937 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3624f424-c602-4bda-b32e-5d92f03a9829 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:47.411225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c823d45-e6a3-4077-a341-deedd5a4636d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:47.411973 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c823d45-e6a3-4077-a341-deedd5a4636d None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:47.412414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c823d45-e6a3-4077-a341-deedd5a4636d None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:47.412741 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c823d45-e6a3-4077-a341-deedd5a4636d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:47.421649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c823d45-e6a3-4077-a341-deedd5a4636d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:47.423208 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c823d45-e6a3-4077-a341-deedd5a4636d None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:47.436645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-5c823d45-e6a3-4077-a341-deedd5a4636d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:47.436645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:47.436645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:47.436645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:47.436645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:47.436645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:47.436645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:47.436645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:47.436645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:47.436645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:47.436645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:47.436645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:47.436645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:16:47.436645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:16:47.436645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:16:47.438504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:16:47.438504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:47.438504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:47.438504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:47.438504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:47.438504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:47.438504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:47.438504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:47.438504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:47.438504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:47.438504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:47.438504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:47.438504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:47.438504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:47.438504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:47.438504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:47.438504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:47.440832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:16:47.440832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:16:47.440832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:16:47.440832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:16:47.440832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:16:47.440832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:16:47.440832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:47.440832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:47.440832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 533/1075] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:16:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:47.445741 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d36607-3856-4097-999b-22626a060120 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:47.494681 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:47.613417 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d36607-3856-4097-999b-22626a060120 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:47.669506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe430b6d-eccd-4b72-8318-94401539b877 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:47.670592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe430b6d-eccd-4b72-8318-94401539b877 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:47.671014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe430b6d-eccd-4b72-8318-94401539b877 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:47.671427 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe430b6d-eccd-4b72-8318-94401539b877 None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:47.672654 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe430b6d-eccd-4b72-8318-94401539b877 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:47.673196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe430b6d-eccd-4b72-8318-94401539b877 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:47.675810 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe430b6d-eccd-4b72-8318-94401539b877 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:47.697808 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 543/1076] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:16:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:47.705279 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c823d45-e6a3-4077-a341-deedd5a4636d None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:47.813851 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:47.827205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c823d45-e6a3-4077-a341-deedd5a4636d None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:47.867780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42996452-8781-4ad5-9461-cca19fb16437 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:47.868489 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42996452-8781-4ad5-9461-cca19fb16437 None admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:47.868773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42996452-8781-4ad5-9461-cca19fb16437 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:47.869048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42996452-8781-4ad5-9461-cca19fb16437 None admin] PATH_INFO: `/v3/projects/895ece8b70ef44adadc3416f81a80731/users/03081d7d581745c88c12e9d089fcb2d3/roles/47e35bb0c3b44184986f4e28032e8c17` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:47.871993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42996452-8781-4ad5-9461-cca19fb16437 None admin] RBAC: Authorizing `identity:create_grant(project_id=895ece8b70ef44adadc3416f81a80731, user_id=03081d7d581745c88c12e9d089fcb2d3, role_id=47e35bb0c3b44184986f4e28032e8c17)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:47.873491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42996452-8781-4ad5-9461-cca19fb16437 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:47.892848 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 534/1077] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:16:47 2022] PUT /identity/v3/projects/895ece8b70ef44adadc3416f81a80731/users/03081d7d581745c88c12e9d089fcb2d3/roles/47e35bb0c3b44184986f4e28032e8c17 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:16:48.613932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bba131e-253d-4bf0-bb79-e9d9623fe67c None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:48.614475 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bba131e-253d-4bf0-bb79-e9d9623fe67c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:48.614836 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bba131e-253d-4bf0-bb79-e9d9623fe67c None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:48.617259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 544/1078] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:48 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:48.626719 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34064b0-ba5d-41a5-ac6c-12ae24d15b8e None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:48.627295 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34064b0-ba5d-41a5-ac6c-12ae24d15b8e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:48.627667 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34064b0-ba5d-41a5-ac6c-12ae24d15b8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:48.659228 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-d34064b0-ba5d-41a5-ac6c-12ae24d15b8e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:48.661358 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d34064b0-ba5d-41a5-ac6c-12ae24d15b8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:48.775600 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 535/1079] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:49.226903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bba131e-253d-4bf0-bb79-e9d9623fe67c None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:49.331535 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:49.357070 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bba131e-253d-4bf0-bb79-e9d9623fe67c None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:49.409664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad72e36-f05d-47e2-8ee7-fbb3b5e14b38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:49.410748 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad72e36-f05d-47e2-8ee7-fbb3b5e14b38 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:49.411243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad72e36-f05d-47e2-8ee7-fbb3b5e14b38 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:49.411732 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad72e36-f05d-47e2-8ee7-fbb3b5e14b38 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:49.431921 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad72e36-f05d-47e2-8ee7-fbb3b5e14b38 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:16:49.442028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-dad72e36-f05d-47e2-8ee7-fbb3b5e14b38 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:16:49.442028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:49.442028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:49.442028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:49.442028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:49.442028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:49.442028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:49.442028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:49.442028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:49.442028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:49.442028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:49.442028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:49.442028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:16:49.442028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:16:49.442028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:16:49.442028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:16:49.444061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:16:49.444061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:16:49.444061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:16:49.444061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:16:49.444061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:49.444061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:49.444061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:49.444061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:49.444061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:49.444061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:49.444061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:49.444061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:49.444061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:49.444061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:49.444061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:49.444061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:49.445924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:49.445924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:49.445924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:49.445924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:49.445924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:16:49.445924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:16:49.445924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:16:49.445924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:16:49.445924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:16:49.445924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:16:49.445924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:16:49.445924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:49.447505 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 545/1080] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:16:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:49.454351 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34064b0-ba5d-41a5-ac6c-12ae24d15b8e None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:49.597791 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:49.630058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34064b0-ba5d-41a5-ac6c-12ae24d15b8e None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:49.669702 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14895b1b-e503-4758-8329-b3363818bcda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:49.670328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14895b1b-e503-4758-8329-b3363818bcda None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:49.670609 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14895b1b-e503-4758-8329-b3363818bcda None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:49.670885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14895b1b-e503-4758-8329-b3363818bcda None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:49.671566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14895b1b-e503-4758-8329-b3363818bcda None admin] RBAC: Adding query filter params (name=admin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:49.671869 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14895b1b-e503-4758-8329-b3363818bcda None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:49.673312 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14895b1b-e503-4758-8329-b3363818bcda None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:49.687758 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 536/1081] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:16:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:49.689507 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad72e36-f05d-47e2-8ee7-fbb3b5e14b38 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:49.799813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad72e36-f05d-47e2-8ee7-fbb3b5e14b38 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:49.837712 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f4ef7c-838c-4a6a-84b6-2f94519e45d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:49.838349 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f4ef7c-838c-4a6a-84b6-2f94519e45d6 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:49.838627 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f4ef7c-838c-4a6a-84b6-2f94519e45d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:49.838927 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f4ef7c-838c-4a6a-84b6-2f94519e45d6 None admin] PATH_INFO: `/v3/users/ca859d18d0da4f85be4a6d6062e5ddb7` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:49.840096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f4ef7c-838c-4a6a-84b6-2f94519e45d6 None admin] RBAC: Authorizing `identity:get_user(user_id=ca859d18d0da4f85be4a6d6062e5ddb7)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:49.841570 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f4ef7c-838c-4a6a-84b6-2f94519e45d6 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:49.843207 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 546/1082] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:16:49 2022] GET /identity/v3/users/ca859d18d0da4f85be4a6d6062e5ddb7 => generated 285 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:49.849325 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14895b1b-e503-4758-8329-b3363818bcda None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:49.962982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14895b1b-e503-4758-8329-b3363818bcda None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:50.000800 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c66ec31-577e-4ad6-8393-90dc0a649bca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:50.001563 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c66ec31-577e-4ad6-8393-90dc0a649bca None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:50.001944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c66ec31-577e-4ad6-8393-90dc0a649bca None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:50.002309 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c66ec31-577e-4ad6-8393-90dc0a649bca None admin] PATH_INFO: `/v3/projects/585f5846d80a486cb51074489e643309` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:50.003591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c66ec31-577e-4ad6-8393-90dc0a649bca None admin] RBAC: Authorizing `identity:get_project(project_id=585f5846d80a486cb51074489e643309)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:50.005160 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c66ec31-577e-4ad6-8393-90dc0a649bca None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:50.006937 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 537/1083] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:16:49 2022] GET /identity/v3/projects/585f5846d80a486cb51074489e643309 => generated 311 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:50.013298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f4ef7c-838c-4a6a-84b6-2f94519e45d6 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:50.025821 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:50.125494 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f4ef7c-838c-4a6a-84b6-2f94519e45d6 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:50.165487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee39c88-a707-43bb-932e-bba22c8db5bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:50.166147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee39c88-a707-43bb-932e-bba22c8db5bc None admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:50.166424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee39c88-a707-43bb-932e-bba22c8db5bc None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:50.166698 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee39c88-a707-43bb-932e-bba22c8db5bc None admin] PATH_INFO: `/v3/projects/585f5846d80a486cb51074489e643309/users/ca859d18d0da4f85be4a6d6062e5ddb7/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:50.169393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee39c88-a707-43bb-932e-bba22c8db5bc None admin] RBAC: Authorizing `identity:create_grant(project_id=585f5846d80a486cb51074489e643309, user_id=ca859d18d0da4f85be4a6d6062e5ddb7, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:50.170835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee39c88-a707-43bb-932e-bba22c8db5bc None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:50.185510 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 547/1084] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:16:50 2022] PUT /identity/v3/projects/585f5846d80a486cb51074489e643309/users/ca859d18d0da4f85be4a6d6062e5ddb7/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:16:50.433966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abf7474-152e-40e0-99a9-0b273e91afcf None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:50.434207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abf7474-152e-40e0-99a9-0b273e91afcf None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:50.434412 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abf7474-152e-40e0-99a9-0b273e91afcf None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:50.435971 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 538/1085] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:16:50.441606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810d53e8-022c-4c6b-a41d-3cc7546a7929 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:50.441963 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810d53e8-022c-4c6b-a41d-3cc7546a7929 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:50.442297 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810d53e8-022c-4c6b-a41d-3cc7546a7929 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:50.462060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-810d53e8-022c-4c6b-a41d-3cc7546a7929 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:50.463260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-810d53e8-022c-4c6b-a41d-3cc7546a7929 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:50.530849 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:50.538096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 548/1086] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:16:50.816693 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abf7474-152e-40e0-99a9-0b273e91afcf None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:50.834885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:50.935453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abf7474-152e-40e0-99a9-0b273e91afcf None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:50.974035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f69d30-2612-491f-9630-db2e9457aad8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:50.974691 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f69d30-2612-491f-9630-db2e9457aad8 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:50.974968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f69d30-2612-491f-9630-db2e9457aad8 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:50.975235 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f69d30-2612-491f-9630-db2e9457aad8 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:50.987984 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f69d30-2612-491f-9630-db2e9457aad8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:16:50.992949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-a1f69d30-2612-491f-9630-db2e9457aad8 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 14:16:50.992949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:50.992949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:50.992949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:50.992949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:50.992949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:50.992949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:50.992949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:50.992949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:50.992949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:50.992949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:50.992949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:50.992949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:16:50.992949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:16:50.992949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:16:50.992949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:16:50.994241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:16:50.994241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:16:50.994241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:16:50.994241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:16:50.994241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:50.994241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:50.994241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:50.994241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:50.994241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:50.994241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:50.994241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:50.994241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:50.994241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:50.994241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:50.994241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:50.994241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:50.996216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:50.996216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:50.996216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:50.996216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:50.996216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:16:50.996216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:16:50.996216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:16:50.996216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:16:50.996216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:16:50.996216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:16:50.996216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 14:16:50.996216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:50.996216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 539/1087] 10.222.0.64 () {56 vars in 1120 bytes} [Tue Oct 4 14:16:50 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:51.000732 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810d53e8-022c-4c6b-a41d-3cc7546a7929 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:51.114709 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810d53e8-022c-4c6b-a41d-3cc7546a7929 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:51.156011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33befd50-c025-4f91-a3ad-33d58e11638e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:51.156739 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33befd50-c025-4f91-a3ad-33d58e11638e None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:51.157084 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33befd50-c025-4f91-a3ad-33d58e11638e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:51.157412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33befd50-c025-4f91-a3ad-33d58e11638e None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:51.158147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33befd50-c025-4f91-a3ad-33d58e11638e None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:51.158503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33befd50-c025-4f91-a3ad-33d58e11638e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:51.159969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33befd50-c025-4f91-a3ad-33d58e11638e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:51.173059 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 549/1088] 10.222.0.64 () {56 vars in 1115 bytes} [Tue Oct 4 14:16:50 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:51.179786 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f69d30-2612-491f-9630-db2e9457aad8 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:51.270666 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:51.320682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f69d30-2612-491f-9630-db2e9457aad8 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:51.377011 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0750fa72-28b1-487f-bce9-931a3581bec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:51.377855 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0750fa72-28b1-487f-bce9-931a3581bec9 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:51.378341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0750fa72-28b1-487f-bce9-931a3581bec9 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:51.378821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0750fa72-28b1-487f-bce9-931a3581bec9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:51.395640 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0750fa72-28b1-487f-bce9-931a3581bec9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:51.397293 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0750fa72-28b1-487f-bce9-931a3581bec9 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:51.414454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-0750fa72-28b1-487f-bce9-931a3581bec9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:51.414454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:51.414454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:51.414454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:51.414454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:51.414454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:51.414454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:51.414454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:51.414454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:51.414454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:51.414454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:51.414454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:51.414454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:51.414454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:51.414454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:51.414454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:51.416467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:51.416467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:51.416467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:51.416467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:51.416467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:51.416467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:51.416467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:51.416467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:51.416467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:51.416467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:51.416467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:51.416467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:51.416467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:51.416467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:51.416467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:51.416467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:51.416467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:51.418638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:51.418638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:51.418638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:51.418638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 540/1089] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:51.422534 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33befd50-c025-4f91-a3ad-33d58e11638e None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:51.532147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:51.608753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33befd50-c025-4f91-a3ad-33d58e11638e None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:51.671423 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d932828b-cf25-4235-9f5e-de242d50c0a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:51.672379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d932828b-cf25-4235-9f5e-de242d50c0a7 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:51.672876 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d932828b-cf25-4235-9f5e-de242d50c0a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:51.673365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d932828b-cf25-4235-9f5e-de242d50c0a7 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:51.674503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d932828b-cf25-4235-9f5e-de242d50c0a7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:51.675035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d932828b-cf25-4235-9f5e-de242d50c0a7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:51.677609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d932828b-cf25-4235-9f5e-de242d50c0a7 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:51.705818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 550/1090] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:51.707169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0750fa72-28b1-487f-bce9-931a3581bec9 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:51.884203 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0750fa72-28b1-487f-bce9-931a3581bec9 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:51.925118 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:51.938356 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368773f9-45eb-48d3-b016-02d82b8fbe83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:51.938996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368773f9-45eb-48d3-b016-02d82b8fbe83 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:51.939276 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368773f9-45eb-48d3-b016-02d82b8fbe83 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:51.939549 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368773f9-45eb-48d3-b016-02d82b8fbe83 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:51.949019 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368773f9-45eb-48d3-b016-02d82b8fbe83 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:51.950937 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368773f9-45eb-48d3-b016-02d82b8fbe83 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:51.963982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-368773f9-45eb-48d3-b016-02d82b8fbe83 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:16:51.963982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:51.963982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:51.963982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:51.963982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:51.963982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:51.963982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:51.963982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:51.963982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:51.963982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:51.963982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:51.963982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:51.963982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:16:51.963982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:16:51.963982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:16:51.963982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:51.966921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:51.969041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:51.969041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:51.969041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:16:51.969041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:16:51.969041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:16:51.969041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:16:51.969041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:16:51.969041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:16:51.969041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 14:16:51.969041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:51.969041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 541/1091] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Oct 4 14:16:51 2022] GET /identity/v3/users/nova => generated 82 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:51.975502 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d932828b-cf25-4235-9f5e-de242d50c0a7 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:52.102040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d932828b-cf25-4235-9f5e-de242d50c0a7 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:52.126936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:52.144490 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb69424-f5d0-478e-9efc-e7b3688ecad2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:52.145103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb69424-f5d0-478e-9efc-e7b3688ecad2 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:52.145374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb69424-f5d0-478e-9efc-e7b3688ecad2 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:52.145641 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb69424-f5d0-478e-9efc-e7b3688ecad2 None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:52.146378 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb69424-f5d0-478e-9efc-e7b3688ecad2 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:52.146677 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb69424-f5d0-478e-9efc-e7b3688ecad2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:52.148153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb69424-f5d0-478e-9efc-e7b3688ecad2 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:52.170865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 551/1092] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Oct 4 14:16:51 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:52.177253 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368773f9-45eb-48d3-b016-02d82b8fbe83 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:52.310206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368773f9-45eb-48d3-b016-02d82b8fbe83 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:52.357366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb4b6a2-7cfd-4780-8bc2-e95b1c74b875 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:52.358127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb4b6a2-7cfd-4780-8bc2-e95b1c74b875 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:52.358497 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb4b6a2-7cfd-4780-8bc2-e95b1c74b875 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:52.358867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb4b6a2-7cfd-4780-8bc2-e95b1c74b875 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:52.430914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:52.454031 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb4b6a2-7cfd-4780-8bc2-e95b1c74b875 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:52.456184 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb4b6a2-7cfd-4780-8bc2-e95b1c74b875 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:52.530566 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad2dfbe-9bcf-4a6b-9f85-26a9233a07bb None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:52.530884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad2dfbe-9bcf-4a6b-9f85-26a9233a07bb None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:52.531107 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad2dfbe-9bcf-4a6b-9f85-26a9233a07bb None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:52.532525 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 552/1093] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:16:52.541098 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32a645d-173d-4e02-8c8a-b0f1740d611f None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:52.541361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32a645d-173d-4e02-8c8a-b0f1740d611f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:52.541635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32a645d-173d-4e02-8c8a-b0f1740d611f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:52.561387 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-f32a645d-173d-4e02-8c8a-b0f1740d611f None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:52.562595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f32a645d-173d-4e02-8c8a-b0f1740d611f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:52.601205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 542/1094] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:16:52 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 04 14:16:52.610398 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb4b6a2-7cfd-4780-8bc2-e95b1c74b875 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:52.647154 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 553/1095] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:16:52.708968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:52.726021 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb4b6a2-7cfd-4780-8bc2-e95b1c74b875 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:52.764115 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed08d781-45ed-466f-9ca1-d05f190124db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:52.764790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed08d781-45ed-466f-9ca1-d05f190124db None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:52.765072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed08d781-45ed-466f-9ca1-d05f190124db None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:52.765349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed08d781-45ed-466f-9ca1-d05f190124db None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:52.777137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed08d781-45ed-466f-9ca1-d05f190124db None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:52.779213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed08d781-45ed-466f-9ca1-d05f190124db None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:52.795810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-ed08d781-45ed-466f-9ca1-d05f190124db None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:52.795810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:52.795810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:52.795810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:52.795810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:52.795810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:52.795810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:52.795810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:52.795810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:52.795810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:52.795810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:52.795810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:52.795810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:16:52.795810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:16:52.795810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:16:52.795810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:16:52.797909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:52.797909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:52.797909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:52.797909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:52.797909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:52.797909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:52.797909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:52.797909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:52.797909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:52.797909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:52.797909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:52.797909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:52.797909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:52.797909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:52.797909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:52.797909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:52.797909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:16:52.799900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:16:52.799900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:16:52.799900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:52.799900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 543/1096] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:52.806711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32a645d-173d-4e02-8c8a-b0f1740d611f None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:52.967785 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32a645d-173d-4e02-8c8a-b0f1740d611f None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:52.978610 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:53.006047 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed08d781-45ed-466f-9ca1-d05f190124db None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:53.023545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5d0dfb-6095-47f9-bee9-563f172f2ea0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:53.024172 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5d0dfb-6095-47f9-bee9-563f172f2ea0 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:53.024495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5d0dfb-6095-47f9-bee9-563f172f2ea0 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:53.024758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5d0dfb-6095-47f9-bee9-563f172f2ea0 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:53.025415 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5d0dfb-6095-47f9-bee9-563f172f2ea0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:53.025716 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5d0dfb-6095-47f9-bee9-563f172f2ea0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:53.028358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5d0dfb-6095-47f9-bee9-563f172f2ea0 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:53.041704 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 554/1097] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:16:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:53.047920 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5d0dfb-6095-47f9-bee9-563f172f2ea0 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:53.148483 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed08d781-45ed-466f-9ca1-d05f190124db None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:53.196726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5d0dfb-6095-47f9-bee9-563f172f2ea0 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:53.200247 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b471ba6-3ef8-4a56-99a0-7fc6442e5562 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:53.200944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b471ba6-3ef8-4a56-99a0-7fc6442e5562 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:53.201228 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b471ba6-3ef8-4a56-99a0-7fc6442e5562 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:53.201511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b471ba6-3ef8-4a56-99a0-7fc6442e5562 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:53.205124 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:53.218549 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b471ba6-3ef8-4a56-99a0-7fc6442e5562 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:16:53.223846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-5b471ba6-3ef8-4a56-99a0-7fc6442e5562 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:16:53.223846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:53.223846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:53.223846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:53.223846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:53.223846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:53.223846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:53.223846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:53.223846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:53.223846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:53.223846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:53.223846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:53.223846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:16:53.223846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:16:53.223846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:16:53.223846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:16:53.225343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:16:53.225343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:16:53.225343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:16:53.225343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:16:53.225343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:53.225343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:53.225343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:53.225343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:53.225343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:53.225343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:53.225343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:53.225343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:53.225343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:53.225343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:53.225343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:53.225343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:53.226461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:53.226461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:53.226461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:53.226461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:53.226461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:16:53.226461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:16:53.226461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:16:53.226461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:16:53.226461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:16:53.226461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:16:53.226461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:16:53.226461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:16:53.226461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 544/1098] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:16:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:16:53.232176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b471ba6-3ef8-4a56-99a0-7fc6442e5562 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:53.253267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b7dab9-37e1-454e-b204-08143d5957da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:53.254077 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b7dab9-37e1-454e-b204-08143d5957da None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:53.254502 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b7dab9-37e1-454e-b204-08143d5957da None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:53.254867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b7dab9-37e1-454e-b204-08143d5957da None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:53.264361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b7dab9-37e1-454e-b204-08143d5957da None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:53.265883 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b7dab9-37e1-454e-b204-08143d5957da None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:53.285424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-37b7dab9-37e1-454e-b204-08143d5957da None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:53.285424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:16:53.285424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:16:53.285424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:16:53.285424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:16:53.285424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:16:53.285424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:16:53.285424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:16:53.285424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:16:53.285424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:16:53.285424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:16:53.285424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:16:53.285424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:16:53.285424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:16:53.285424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:16:53.287466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:16:53.287466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:16:53.287466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:16:53.287466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:16:53.287466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:16:53.287466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:16:53.287466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:16:53.287466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:16:53.287466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:16:53.287466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:16:53.287466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:16:53.287466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:16:53.287466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:16:53.287466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:16:53.287466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:16:53.287466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:16:53.287466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:16:53.291597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:16:53.291597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:16:53.291597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:16:53.291597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:16:53.291597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:16:53.291597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:16:53.291597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:16:53.291597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:16:53.291597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 555/1099] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:16:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:16:53.294145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b7dab9-37e1-454e-b204-08143d5957da None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:53.339515 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:53.364870 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b471ba6-3ef8-4a56-99a0-7fc6442e5562 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:53.418717 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8205e33-8494-468e-b413-b6a1f54389f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:53.419769 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8205e33-8494-468e-b413-b6a1f54389f3 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:53.420210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8205e33-8494-468e-b413-b6a1f54389f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:53.420527 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8205e33-8494-468e-b413-b6a1f54389f3 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:53.421389 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8205e33-8494-468e-b413-b6a1f54389f3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:53.421732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8205e33-8494-468e-b413-b6a1f54389f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:53.423842 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8205e33-8494-468e-b413-b6a1f54389f3 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:53.446469 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 545/1100] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:16:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:53.451191 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8205e33-8494-468e-b413-b6a1f54389f3 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:53.455981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b7dab9-37e1-454e-b204-08143d5957da None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:53.508032 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05212495-c1e7-448a-bae8-86e5004bcbc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:53.508926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05212495-c1e7-448a-bae8-86e5004bcbc3 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:53.509373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05212495-c1e7-448a-bae8-86e5004bcbc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:53.509818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05212495-c1e7-448a-bae8-86e5004bcbc3 None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:53.510951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05212495-c1e7-448a-bae8-86e5004bcbc3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:53.511473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05212495-c1e7-448a-bae8-86e5004bcbc3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:53.513468 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05212495-c1e7-448a-bae8-86e5004bcbc3 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:53.537128 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 556/1101] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:16:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:53.546087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05212495-c1e7-448a-bae8-86e5004bcbc3 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:53.552824 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:53.604143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8205e33-8494-468e-b413-b6a1f54389f3 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:53.658293 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d20c92-04e4-47d7-bcce-6d5d9b5219dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:53.659028 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d20c92-04e4-47d7-bcce-6d5d9b5219dc None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:53.659382 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d20c92-04e4-47d7-bcce-6d5d9b5219dc None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:53.659734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d20c92-04e4-47d7-bcce-6d5d9b5219dc None admin] PATH_INFO: `/v3/users/ca859d18d0da4f85be4a6d6062e5ddb7` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:53.661240 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d20c92-04e4-47d7-bcce-6d5d9b5219dc None admin] RBAC: Authorizing `identity:get_user(user_id=ca859d18d0da4f85be4a6d6062e5ddb7)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:53.663142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d20c92-04e4-47d7-bcce-6d5d9b5219dc None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:53.665026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 546/1102] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:16:53 2022] GET /identity/v3/users/ca859d18d0da4f85be4a6d6062e5ddb7 => generated 285 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:53.669831 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:53.673145 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d20c92-04e4-47d7-bcce-6d5d9b5219dc None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:53.687598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05212495-c1e7-448a-bae8-86e5004bcbc3 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:53.734023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9995ad06-acfe-4d6f-937c-98ce8ba6a2f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:53.734760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995ad06-acfe-4d6f-937c-98ce8ba6a2f7 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:53.735100 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995ad06-acfe-4d6f-937c-98ce8ba6a2f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:53.735492 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995ad06-acfe-4d6f-937c-98ce8ba6a2f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:53.736796 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9995ad06-acfe-4d6f-937c-98ce8ba6a2f7 None admin] RBAC: Adding query filter params (role.id=47e35bb0c3b44184986f4e28032e8c17, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=03081d7d581745c88c12e9d089fcb2d3) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:53.736999 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9995ad06-acfe-4d6f-937c-98ce8ba6a2f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:53.738543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9995ad06-acfe-4d6f-937c-98ce8ba6a2f7 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:53.747209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 557/1103] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:16:53 2022] GET /identity/v3/role_assignments?user.id=03081d7d581745c88c12e9d089fcb2d3&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=47e35bb0c3b44184986f4e28032e8c17 => generated 608 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:53.803516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d20c92-04e4-47d7-bcce-6d5d9b5219dc None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:53.838125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:53.850460 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a0e5c7-6265-45d0-8a32-f67b6e4b59a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:53.851173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a0e5c7-6265-45d0-8a32-f67b6e4b59a2 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:53.851516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a0e5c7-6265-45d0-8a32-f67b6e4b59a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:53.851856 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a0e5c7-6265-45d0-8a32-f67b6e4b59a2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:53.940871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a0e5c7-6265-45d0-8a32-f67b6e4b59a2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:53.942364 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a0e5c7-6265-45d0-8a32-f67b6e4b59a2 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:54.086647 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 547/1104] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:16:53 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 04 14:16:54.096258 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9995ad06-acfe-4d6f-937c-98ce8ba6a2f7 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:54.233531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9995ad06-acfe-4d6f-937c-98ce8ba6a2f7 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:54.277725 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db302330-737d-4e0a-b71c-c56a95db8f95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:54.278635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db302330-737d-4e0a-b71c-c56a95db8f95 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:54.278907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db302330-737d-4e0a-b71c-c56a95db8f95 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:54.279174 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db302330-737d-4e0a-b71c-c56a95db8f95 None admin] PATH_INFO: `/v3/projects/585f5846d80a486cb51074489e643309` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:54.280472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db302330-737d-4e0a-b71c-c56a95db8f95 None admin] RBAC: Authorizing `identity:get_project(project_id=585f5846d80a486cb51074489e643309)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:54.281953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db302330-737d-4e0a-b71c-c56a95db8f95 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:54.283728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 558/1105] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:16:54 2022] GET /identity/v3/projects/585f5846d80a486cb51074489e643309 => generated 311 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:54.290605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a0e5c7-6265-45d0-8a32-f67b6e4b59a2 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:54.325606 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:54.421106 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a0e5c7-6265-45d0-8a32-f67b6e4b59a2 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:54.469232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf2f937-5a12-48a5-81c3-fae10d120c5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:54.469849 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf2f937-5a12-48a5-81c3-fae10d120c5d None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:54.470125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf2f937-5a12-48a5-81c3-fae10d120c5d None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:54.470395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf2f937-5a12-48a5-81c3-fae10d120c5d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:54.471133 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf2f937-5a12-48a5-81c3-fae10d120c5d None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=585f5846d80a486cb51074489e643309, user.id=ca859d18d0da4f85be4a6d6062e5ddb7) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:16:54.471442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf2f937-5a12-48a5-81c3-fae10d120c5d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:54.472964 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf2f937-5a12-48a5-81c3-fae10d120c5d None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:54.482962 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 548/1106] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:16:54 2022] GET /identity/v3/role_assignments?user.id=ca859d18d0da4f85be4a6d6062e5ddb7&scope.project.id=585f5846d80a486cb51074489e643309&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 608 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:16:56.832439 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674daece-8af8-4a4d-9b7c-5f969712db17 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:56.833260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674daece-8af8-4a4d-9b7c-5f969712db17 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:56.833260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674daece-8af8-4a4d-9b7c-5f969712db17 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:56.834374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 559/1107] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:16:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:16:56.841382 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf338b9-ef8f-4d0d-80ac-2e1757ec0cb0 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:56.841961 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf338b9-ef8f-4d0d-80ac-2e1757ec0cb0 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:56.841961 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf338b9-ef8f-4d0d-80ac-2e1757ec0cb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:56.861438 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-3bf338b9-ef8f-4d0d-80ac-2e1757ec0cb0 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:16:56.862671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-3bf338b9-ef8f-4d0d-80ac-2e1757ec0cb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:16:56.946543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 549/1108] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:16:56 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:16:57.227317 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674daece-8af8-4a4d-9b7c-5f969712db17 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:57.259957 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:57.363314 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674daece-8af8-4a4d-9b7c-5f969712db17 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:57.403537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec90cc16-5864-40e5-924e-243ab29c6b8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:57.404189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec90cc16-5864-40e5-924e-243ab29c6b8a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:57.404503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec90cc16-5864-40e5-924e-243ab29c6b8a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:57.404773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec90cc16-5864-40e5-924e-243ab29c6b8a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:57.405991 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec90cc16-5864-40e5-924e-243ab29c6b8a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:57.407441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec90cc16-5864-40e5-924e-243ab29c6b8a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:57.409142 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 560/1109] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:16:57 2022] GET /identity/v3/domains/default => generated 199 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:16:57.415511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf338b9-ef8f-4d0d-80ac-2e1757ec0cb0 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:16:57.440709 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:16:57.532409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf338b9-ef8f-4d0d-80ac-2e1757ec0cb0 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:16:57.571001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068e4f0a-9295-4fe0-b9fa-70092920bc35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:16:57.571626 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068e4f0a-9295-4fe0-b9fa-70092920bc35 None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:16:57.571907 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068e4f0a-9295-4fe0-b9fa-70092920bc35 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:16:57.572181 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068e4f0a-9295-4fe0-b9fa-70092920bc35 None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:16:57.573015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068e4f0a-9295-4fe0-b9fa-70092920bc35 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:16:57.574478 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068e4f0a-9295-4fe0-b9fa-70092920bc35 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:16:57.598477 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 550/1110] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:16:57 2022] POST /identity/v3/users => generated 286 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:17:00.022789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fd92ed-8b5e-4419-9e06-c2df045c8f15 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:00.023390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fd92ed-8b5e-4419-9e06-c2df045c8f15 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:00.023390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fd92ed-8b5e-4419-9e06-c2df045c8f15 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:00.024921 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 561/1111] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:17:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:17:00.032193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2be340-1c10-45a3-9fe0-2ec6d1882f82 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:00.032852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2be340-1c10-45a3-9fe0-2ec6d1882f82 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:00.032852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2be340-1c10-45a3-9fe0-2ec6d1882f82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:00.056385 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-fa2be340-1c10-45a3-9fe0-2ec6d1882f82 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:17:00.057660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-fa2be340-1c10-45a3-9fe0-2ec6d1882f82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:17:00.139245 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 551/1112] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:17:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:17:00.584573 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fd92ed-8b5e-4419-9e06-c2df045c8f15 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:00.691675 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:00.707339 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fd92ed-8b5e-4419-9e06-c2df045c8f15 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:00.746441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662f4ffd-4ec3-44f3-941e-201643ba3e28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:00.747158 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662f4ffd-4ec3-44f3-941e-201643ba3e28 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:00.747450 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662f4ffd-4ec3-44f3-941e-201643ba3e28 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:00.747802 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662f4ffd-4ec3-44f3-941e-201643ba3e28 None admin] PATH_INFO: `/v3/roles/e114d28e2dd147fcb39452dd461791c0` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:00.749425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662f4ffd-4ec3-44f3-941e-201643ba3e28 None admin] RBAC: Authorizing `identity:get_role(role_id=e114d28e2dd147fcb39452dd461791c0)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:00.750979 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662f4ffd-4ec3-44f3-941e-201643ba3e28 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:00.752650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 562/1113] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:17:00 2022] GET /identity/v3/roles/e114d28e2dd147fcb39452dd461791c0 => generated 223 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:17:00.763277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2be340-1c10-45a3-9fe0-2ec6d1882f82 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:00.778904 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:00.891584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2be340-1c10-45a3-9fe0-2ec6d1882f82 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:00.929800 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beee4de-0c15-4124-baf8-3aceb2e547b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:00.930443 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beee4de-0c15-4124-baf8-3aceb2e547b0 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:00.930726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beee4de-0c15-4124-baf8-3aceb2e547b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:00.931000 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beee4de-0c15-4124-baf8-3aceb2e547b0 None admin] PATH_INFO: `/v3/users/cdfd508cbd2d4b609ebe88238f3deb01` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:00.947115 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6beee4de-0c15-4124-baf8-3aceb2e547b0 None admin] RBAC: Authorizing `identity:get_user(user_id=cdfd508cbd2d4b609ebe88238f3deb01)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:00.949288 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6beee4de-0c15-4124-baf8-3aceb2e547b0 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:00.951239 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 552/1114] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:17:00 2022] GET /identity/v3/users/cdfd508cbd2d4b609ebe88238f3deb01 => generated 286 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:00.959231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662f4ffd-4ec3-44f3-941e-201643ba3e28 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:01.092540 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662f4ffd-4ec3-44f3-941e-201643ba3e28 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:01.167420 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a40442c-c888-4bfa-b918-f893febe951f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:01.168486 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a40442c-c888-4bfa-b918-f893febe951f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:01.168970 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a40442c-c888-4bfa-b918-f893febe951f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:01.169440 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a40442c-c888-4bfa-b918-f893febe951f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:01.216585 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:01.269837 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a40442c-c888-4bfa-b918-f893febe951f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:01.271303 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a40442c-c888-4bfa-b918-f893febe951f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:01.386574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 563/1115] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:17:00 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 14:17:01.393014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beee4de-0c15-4124-baf8-3aceb2e547b0 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:01.471511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:01.513057 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beee4de-0c15-4124-baf8-3aceb2e547b0 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:01.551136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd09c607-80ef-49f3-807d-ce8581c2b5a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:01.551784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd09c607-80ef-49f3-807d-ce8581c2b5a6 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:01.552070 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd09c607-80ef-49f3-807d-ce8581c2b5a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:01.552377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd09c607-80ef-49f3-807d-ce8581c2b5a6 None admin] PATH_INFO: `/v3/projects/585f5846d80a486cb51074489e643309` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:01.553575 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd09c607-80ef-49f3-807d-ce8581c2b5a6 None admin] RBAC: Authorizing `identity:get_project(project_id=585f5846d80a486cb51074489e643309)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:01.555037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd09c607-80ef-49f3-807d-ce8581c2b5a6 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:01.556781 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 553/1116] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:17:01 2022] GET /identity/v3/projects/585f5846d80a486cb51074489e643309 => generated 311 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:17:01.563672 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a40442c-c888-4bfa-b918-f893febe951f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:01.623378 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:01.672687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a40442c-c888-4bfa-b918-f893febe951f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:01.711308 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd66aa1-c5b4-4a05-be79-b70c8b22e6cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:01.711911 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd66aa1-c5b4-4a05-be79-b70c8b22e6cf None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:01.712190 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd66aa1-c5b4-4a05-be79-b70c8b22e6cf None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:01.712494 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd66aa1-c5b4-4a05-be79-b70c8b22e6cf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:01.713228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd66aa1-c5b4-4a05-be79-b70c8b22e6cf None admin] RBAC: Adding query filter params (role.id=e114d28e2dd147fcb39452dd461791c0, scope.project.id=585f5846d80a486cb51074489e643309, user.id=cdfd508cbd2d4b609ebe88238f3deb01) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:17:01.713552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd66aa1-c5b4-4a05-be79-b70c8b22e6cf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:01.715102 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd66aa1-c5b4-4a05-be79-b70c8b22e6cf None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:01.721267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 564/1117] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:17:01 2022] GET /identity/v3/role_assignments?user.id=cdfd508cbd2d4b609ebe88238f3deb01&scope.project.id=585f5846d80a486cb51074489e643309&role.id=e114d28e2dd147fcb39452dd461791c0 => generated 259 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:04.080541 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faac13e-5d14-4626-a368-1ce8959b4108 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:04.081173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faac13e-5d14-4626-a368-1ce8959b4108 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:04.081173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faac13e-5d14-4626-a368-1ce8959b4108 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:04.082456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 554/1118] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:17:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:17:04.088635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4230bb9-9916-45e0-a4f8-7a6d1697c6d3 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:04.089186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4230bb9-9916-45e0-a4f8-7a6d1697c6d3 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:04.089286 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4230bb9-9916-45e0-a4f8-7a6d1697c6d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:04.109180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-e4230bb9-9916-45e0-a4f8-7a6d1697c6d3 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:17:04.110396 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-e4230bb9-9916-45e0-a4f8-7a6d1697c6d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:17:04.185102 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 565/1119] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:17:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:17:04.469004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faac13e-5d14-4626-a368-1ce8959b4108 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:04.590131 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faac13e-5d14-4626-a368-1ce8959b4108 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:04.626922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:04.633130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606ab488-1a6f-4f4b-b6f4-5c85c37f36d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:04.633787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606ab488-1a6f-4f4b-b6f4-5c85c37f36d6 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:04.634068 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606ab488-1a6f-4f4b-b6f4-5c85c37f36d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:04.634343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606ab488-1a6f-4f4b-b6f4-5c85c37f36d6 None admin] PATH_INFO: `/v3/roles/e114d28e2dd147fcb39452dd461791c0` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:04.636440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606ab488-1a6f-4f4b-b6f4-5c85c37f36d6 None admin] RBAC: Authorizing `identity:get_role(role_id=e114d28e2dd147fcb39452dd461791c0)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:04.637850 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606ab488-1a6f-4f4b-b6f4-5c85c37f36d6 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:04.639260 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 555/1120] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:17:04 2022] GET /identity/v3/roles/e114d28e2dd147fcb39452dd461791c0 => generated 223 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:17:04.645462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4230bb9-9916-45e0-a4f8-7a6d1697c6d3 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:04.702007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:04.758673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4230bb9-9916-45e0-a4f8-7a6d1697c6d3 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:04.797706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791e4d36-9f81-4fc0-a7fa-6f0e5f70abbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:04.798347 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791e4d36-9f81-4fc0-a7fa-6f0e5f70abbd None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:04.798630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791e4d36-9f81-4fc0-a7fa-6f0e5f70abbd None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:04.798903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791e4d36-9f81-4fc0-a7fa-6f0e5f70abbd None admin] PATH_INFO: `/v3/users/cdfd508cbd2d4b609ebe88238f3deb01` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:04.800169 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791e4d36-9f81-4fc0-a7fa-6f0e5f70abbd None admin] RBAC: Authorizing `identity:get_user(user_id=cdfd508cbd2d4b609ebe88238f3deb01)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:04.801653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791e4d36-9f81-4fc0-a7fa-6f0e5f70abbd None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:04.803407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 566/1121] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:17:04 2022] GET /identity/v3/users/cdfd508cbd2d4b609ebe88238f3deb01 => generated 286 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:04.811105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606ab488-1a6f-4f4b-b6f4-5c85c37f36d6 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:04.926116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606ab488-1a6f-4f4b-b6f4-5c85c37f36d6 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:04.962821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49e3ddc-71b9-41fa-a029-dce8846d9b24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:04.964763 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49e3ddc-71b9-41fa-a029-dce8846d9b24 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:04.965052 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49e3ddc-71b9-41fa-a029-dce8846d9b24 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:04.965355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49e3ddc-71b9-41fa-a029-dce8846d9b24 None admin] PATH_INFO: `/v3/projects/585f5846d80a486cb51074489e643309` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:04.966635 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49e3ddc-71b9-41fa-a029-dce8846d9b24 None admin] RBAC: Authorizing `identity:get_project(project_id=585f5846d80a486cb51074489e643309)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:04.968111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49e3ddc-71b9-41fa-a029-dce8846d9b24 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:04.969877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 556/1122] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:17:04 2022] GET /identity/v3/projects/585f5846d80a486cb51074489e643309 => generated 311 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:04.977133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791e4d36-9f81-4fc0-a7fa-6f0e5f70abbd None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:05.153876 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791e4d36-9f81-4fc0-a7fa-6f0e5f70abbd None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:05.195662 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:05.225612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb10ed81-a1bd-46bf-ae53-5d4f9d961b5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:05.226702 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb10ed81-a1bd-46bf-ae53-5d4f9d961b5b None admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:05.227196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb10ed81-a1bd-46bf-ae53-5d4f9d961b5b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:05.227687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb10ed81-a1bd-46bf-ae53-5d4f9d961b5b None admin] PATH_INFO: `/v3/projects/585f5846d80a486cb51074489e643309/users/cdfd508cbd2d4b609ebe88238f3deb01/roles/e114d28e2dd147fcb39452dd461791c0` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:05.232057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb10ed81-a1bd-46bf-ae53-5d4f9d961b5b None admin] RBAC: Authorizing `identity:create_grant(project_id=585f5846d80a486cb51074489e643309, user_id=cdfd508cbd2d4b609ebe88238f3deb01, role_id=e114d28e2dd147fcb39452dd461791c0)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:05.234767 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb10ed81-a1bd-46bf-ae53-5d4f9d961b5b None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:05.252934 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 567/1123] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:17:04 2022] PUT /identity/v3/projects/585f5846d80a486cb51074489e643309/users/cdfd508cbd2d4b609ebe88238f3deb01/roles/e114d28e2dd147fcb39452dd461791c0 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:17:07.559436 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2939de4-43f0-456c-9901-e04dbb878527 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:07.560138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2939de4-43f0-456c-9901-e04dbb878527 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:07.560138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2939de4-43f0-456c-9901-e04dbb878527 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:07.561517 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 557/1124] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:17:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:17:07.567718 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8940ef4-6afc-4bd3-8c5f-14d0c38cde3e None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:07.568273 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8940ef4-6afc-4bd3-8c5f-14d0c38cde3e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:07.568273 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8940ef4-6afc-4bd3-8c5f-14d0c38cde3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:07.588560 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-b8940ef4-6afc-4bd3-8c5f-14d0c38cde3e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:17:07.590057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b8940ef4-6afc-4bd3-8c5f-14d0c38cde3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:17:07.669493 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 568/1125] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:17:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:17:07.951573 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2939de4-43f0-456c-9901-e04dbb878527 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:08.029604 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:08.091379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2939de4-43f0-456c-9901-e04dbb878527 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:08.156787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75b5950-9838-4b8d-93a5-75217de6337e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:08.157481 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75b5950-9838-4b8d-93a5-75217de6337e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:08.157768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75b5950-9838-4b8d-93a5-75217de6337e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:08.158047 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75b5950-9838-4b8d-93a5-75217de6337e None admin] PATH_INFO: `/v3/roles/e114d28e2dd147fcb39452dd461791c0` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:08.159585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75b5950-9838-4b8d-93a5-75217de6337e None admin] RBAC: Authorizing `identity:get_role(role_id=e114d28e2dd147fcb39452dd461791c0)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:08.161032 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75b5950-9838-4b8d-93a5-75217de6337e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:08.162422 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 558/1126] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:17:07 2022] GET /identity/v3/roles/e114d28e2dd147fcb39452dd461791c0 => generated 223 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:08.168821 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8940ef4-6afc-4bd3-8c5f-14d0c38cde3e None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:08.287450 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8940ef4-6afc-4bd3-8c5f-14d0c38cde3e None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:08.299415 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:08.331061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df37f91-1b93-4a53-be52-47db1846be39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:08.331762 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df37f91-1b93-4a53-be52-47db1846be39 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:08.332036 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df37f91-1b93-4a53-be52-47db1846be39 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:08.332302 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df37f91-1b93-4a53-be52-47db1846be39 None admin] PATH_INFO: `/v3/users/cdfd508cbd2d4b609ebe88238f3deb01` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:08.333505 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df37f91-1b93-4a53-be52-47db1846be39 None admin] RBAC: Authorizing `identity:get_user(user_id=cdfd508cbd2d4b609ebe88238f3deb01)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:08.334968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df37f91-1b93-4a53-be52-47db1846be39 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:08.336648 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 569/1127] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:17:08 2022] GET /identity/v3/users/cdfd508cbd2d4b609ebe88238f3deb01 => generated 286 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:08.343867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75b5950-9838-4b8d-93a5-75217de6337e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:08.461640 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75b5950-9838-4b8d-93a5-75217de6337e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:08.501075 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094d5e9c-a40b-4cde-9655-f7fdb0a72a7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:08.501698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094d5e9c-a40b-4cde-9655-f7fdb0a72a7c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:08.501986 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094d5e9c-a40b-4cde-9655-f7fdb0a72a7c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:08.502261 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094d5e9c-a40b-4cde-9655-f7fdb0a72a7c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:08.506436 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:08.583543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-094d5e9c-a40b-4cde-9655-f7fdb0a72a7c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:08.584966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-094d5e9c-a40b-4cde-9655-f7fdb0a72a7c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:08.704055 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 559/1128] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:17:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 04 14:17:08.712034 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df37f91-1b93-4a53-be52-47db1846be39 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:08.833531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df37f91-1b93-4a53-be52-47db1846be39 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:08.876404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1468c1cd-ed93-4989-94b4-250685f4269d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:08.877117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1468c1cd-ed93-4989-94b4-250685f4269d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:08.877435 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1468c1cd-ed93-4989-94b4-250685f4269d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:08.877748 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1468c1cd-ed93-4989-94b4-250685f4269d None admin] PATH_INFO: `/v3/projects/585f5846d80a486cb51074489e643309` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:08.879150 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1468c1cd-ed93-4989-94b4-250685f4269d None admin] RBAC: Authorizing `identity:get_project(project_id=585f5846d80a486cb51074489e643309)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:08.880717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1468c1cd-ed93-4989-94b4-250685f4269d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:08.882562 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 570/1129] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:17:08 2022] GET /identity/v3/projects/585f5846d80a486cb51074489e643309 => generated 311 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:08.890544 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094d5e9c-a40b-4cde-9655-f7fdb0a72a7c None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:08.934000 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:09.019697 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094d5e9c-a40b-4cde-9655-f7fdb0a72a7c None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:09.062137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d582372-b63b-47a1-94e2-041c005a1af7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:09.062896 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d582372-b63b-47a1-94e2-041c005a1af7 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:09.063268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d582372-b63b-47a1-94e2-041c005a1af7 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:09.063636 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d582372-b63b-47a1-94e2-041c005a1af7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:09.064558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d582372-b63b-47a1-94e2-041c005a1af7 None admin] RBAC: Adding query filter params (role.id=e114d28e2dd147fcb39452dd461791c0, scope.project.id=585f5846d80a486cb51074489e643309, user.id=cdfd508cbd2d4b609ebe88238f3deb01) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:17:09.064967 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d582372-b63b-47a1-94e2-041c005a1af7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:09.066562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d582372-b63b-47a1-94e2-041c005a1af7 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:09.074090 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 560/1130] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:17:08 2022] GET /identity/v3/role_assignments?user.id=cdfd508cbd2d4b609ebe88238f3deb01&scope.project.id=585f5846d80a486cb51074489e643309&role.id=e114d28e2dd147fcb39452dd461791c0 => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:11.406670 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f51aef-2755-4a66-883e-2a86e73dbd5e None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:11.407229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f51aef-2755-4a66-883e-2a86e73dbd5e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:11.407229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f51aef-2755-4a66-883e-2a86e73dbd5e None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:11.408624 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 571/1131] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:17:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:17:11.414801 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da368221-e5ed-4cbd-a8ad-057c82604944 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:11.415413 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da368221-e5ed-4cbd-a8ad-057c82604944 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:11.415413 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da368221-e5ed-4cbd-a8ad-057c82604944 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:11.436652 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-da368221-e5ed-4cbd-a8ad-057c82604944 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:17:11.437919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-da368221-e5ed-4cbd-a8ad-057c82604944 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:17:11.543366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 561/1132] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:17:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:17:11.853260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f51aef-2755-4a66-883e-2a86e73dbd5e None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:11.929141 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:12.017088 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f51aef-2755-4a66-883e-2a86e73dbd5e None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:12.071992 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89583640-f8ba-4a26-9c6b-3c128d97ca82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:12.072889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89583640-f8ba-4a26-9c6b-3c128d97ca82 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:12.073375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89583640-f8ba-4a26-9c6b-3c128d97ca82 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:12.073723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89583640-f8ba-4a26-9c6b-3c128d97ca82 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:12.075228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89583640-f8ba-4a26-9c6b-3c128d97ca82 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:12.077005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89583640-f8ba-4a26-9c6b-3c128d97ca82 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:12.079837 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 572/1133] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:17:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:12.087170 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da368221-e5ed-4cbd-a8ad-057c82604944 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:12.131576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:12.255301 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da368221-e5ed-4cbd-a8ad-057c82604944 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:12.307925 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38be214-5de8-42a7-9f9d-93616d5094e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:12.308673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38be214-5de8-42a7-9f9d-93616d5094e5 None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:12.308955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38be214-5de8-42a7-9f9d-93616d5094e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:12.309228 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38be214-5de8-42a7-9f9d-93616d5094e5 None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:12.309915 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38be214-5de8-42a7-9f9d-93616d5094e5 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:12.311377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38be214-5de8-42a7-9f9d-93616d5094e5 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:12.352419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 562/1134] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Oct 4 14:17:12 2022] POST /identity/v3/projects => generated 311 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:17:14.769869 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568739da-01a9-4f5c-b5aa-600b7e131f07 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:14.770409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568739da-01a9-4f5c-b5aa-600b7e131f07 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:14.770409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568739da-01a9-4f5c-b5aa-600b7e131f07 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:14.771821 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 573/1135] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:17:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:17:14.778536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8407c2f-c73d-4d0f-9198-1f633d962811 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:14.779075 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8407c2f-c73d-4d0f-9198-1f633d962811 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:14.779075 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8407c2f-c73d-4d0f-9198-1f633d962811 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:14.799061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-a8407c2f-c73d-4d0f-9198-1f633d962811 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:17:14.800363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a8407c2f-c73d-4d0f-9198-1f633d962811 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:17:14.880675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 563/1136] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:17:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:17:15.170334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568739da-01a9-4f5c-b5aa-600b7e131f07 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:15.284670 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568739da-01a9-4f5c-b5aa-600b7e131f07 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:15.302590 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:15.329081 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35de05b1-87e7-4553-8c79-7285c6601ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:15.329714 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35de05b1-87e7-4553-8c79-7285c6601ce7 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:15.329986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35de05b1-87e7-4553-8c79-7285c6601ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:15.330254 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35de05b1-87e7-4553-8c79-7285c6601ce7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:15.331377 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35de05b1-87e7-4553-8c79-7285c6601ce7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:15.332852 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35de05b1-87e7-4553-8c79-7285c6601ce7 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:15.334488 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 574/1137] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:17:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:17:15.340806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8407c2f-c73d-4d0f-9198-1f633d962811 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:15.357395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:15.457046 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8407c2f-c73d-4d0f-9198-1f633d962811 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:15.494626 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008330f0-add4-43ff-9aef-beaab78a9bf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:15.495438 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008330f0-add4-43ff-9aef-beaab78a9bf5 None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:15.495609 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008330f0-add4-43ff-9aef-beaab78a9bf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:15.495883 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008330f0-add4-43ff-9aef-beaab78a9bf5 None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:15.496595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008330f0-add4-43ff-9aef-beaab78a9bf5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:15.498142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008330f0-add4-43ff-9aef-beaab78a9bf5 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:15.522321 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 564/1138] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:17:15 2022] POST /identity/v3/users => generated 286 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:17:17.939872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248bcbbb-cc06-4f8f-bdcf-94b0e58f9716 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:17.940724 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248bcbbb-cc06-4f8f-bdcf-94b0e58f9716 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:17.940724 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248bcbbb-cc06-4f8f-bdcf-94b0e58f9716 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:17.941946 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 575/1139] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:17:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:17:17.948486 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fae6b22-14bb-4281-bd95-5fe83339bc73 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:17.949020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fae6b22-14bb-4281-bd95-5fe83339bc73 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:17.949178 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fae6b22-14bb-4281-bd95-5fe83339bc73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:17.969532 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-4fae6b22-14bb-4281-bd95-5fe83339bc73 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:17:17.970791 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-4fae6b22-14bb-4281-bd95-5fe83339bc73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:17:18.049434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:18.057193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 565/1140] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:17:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:17:18.331417 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248bcbbb-cc06-4f8f-bdcf-94b0e58f9716 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:18.448167 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248bcbbb-cc06-4f8f-bdcf-94b0e58f9716 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:18.485945 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f716f124-8efa-4960-b8d8-dbaefe40c25b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:18.486673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f716f124-8efa-4960-b8d8-dbaefe40c25b None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:18.486992 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f716f124-8efa-4960-b8d8-dbaefe40c25b None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:18.487301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f716f124-8efa-4960-b8d8-dbaefe40c25b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:18.499099 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f716f124-8efa-4960-b8d8-dbaefe40c25b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:17:18.504118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-f716f124-8efa-4960-b8d8-dbaefe40c25b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:17:18.504118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:17:18.504118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:17:18.504118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:17:18.504118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:17:18.504118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:17:18.504118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:17:18.504118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:17:18.504118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:17:18.504118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:17:18.504118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:17:18.504118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:17:18.504118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:17:18.504118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:17:18.504118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:17:18.504118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:17:18.505889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:17:18.505889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:17:18.505889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:17:18.505889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:17:18.505889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:17:18.505889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:17:18.505889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:17:18.505889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:17:18.505889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:17:18.505889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:17:18.505889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:17:18.505889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:17:18.505889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:17:18.505889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:17:18.505889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:17:18.505889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:17:18.507607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:17:18.507607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:17:18.507607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:17:18.507607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:17:18.507607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:17:18.507607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:17:18.507607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:17:18.507607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:17:18.507607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:17:18.507607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:17:18.507607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:17:18.507607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:17:18.507607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 576/1141] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:17:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:17:18.513813 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fae6b22-14bb-4281-bd95-5fe83339bc73 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:18.697456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fae6b22-14bb-4281-bd95-5fe83339bc73 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:18.758072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9a5d8c-e7eb-4256-9401-8a3a8fd988c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:18.758938 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9a5d8c-e7eb-4256-9401-8a3a8fd988c3 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:18.759310 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9a5d8c-e7eb-4256-9401-8a3a8fd988c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:18.759599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9a5d8c-e7eb-4256-9401-8a3a8fd988c3 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:18.760913 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9a5d8c-e7eb-4256-9401-8a3a8fd988c3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:17:18.761442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9a5d8c-e7eb-4256-9401-8a3a8fd988c3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:18.763687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9a5d8c-e7eb-4256-9401-8a3a8fd988c3 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:18.780166 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 566/1142] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:17:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:18.787017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f716f124-8efa-4960-b8d8-dbaefe40c25b None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:18.838987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:18.909453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f716f124-8efa-4960-b8d8-dbaefe40c25b None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:18.951119 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f61225-5114-401d-bde1-8a2ce0bfef29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:18.951763 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f61225-5114-401d-bde1-8a2ce0bfef29 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:18.951936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f61225-5114-401d-bde1-8a2ce0bfef29 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:18.952209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f61225-5114-401d-bde1-8a2ce0bfef29 None admin] PATH_INFO: `/v3/users/d477c8a041ad4fa78c6f8aa50411137c` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:18.967381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f61225-5114-401d-bde1-8a2ce0bfef29 None admin] RBAC: Authorizing `identity:get_user(user_id=d477c8a041ad4fa78c6f8aa50411137c)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:18.968893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f61225-5114-401d-bde1-8a2ce0bfef29 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:18.970647 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 577/1143] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:17:18 2022] GET /identity/v3/users/d477c8a041ad4fa78c6f8aa50411137c => generated 286 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:18.976828 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9a5d8c-e7eb-4256-9401-8a3a8fd988c3 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:19.043672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:19.092283 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9a5d8c-e7eb-4256-9401-8a3a8fd988c3 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:19.128930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e800646-9a61-4b4d-b017-7de950398b1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:19.129550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e800646-9a61-4b4d-b017-7de950398b1e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:19.129835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e800646-9a61-4b4d-b017-7de950398b1e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:19.130112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e800646-9a61-4b4d-b017-7de950398b1e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:19.206238 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e800646-9a61-4b4d-b017-7de950398b1e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:19.207625 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e800646-9a61-4b4d-b017-7de950398b1e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:19.270648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:19.317125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 567/1144] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:17:18 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 14:17:19.324195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f61225-5114-401d-bde1-8a2ce0bfef29 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:19.447392 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:19.458876 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f61225-5114-401d-bde1-8a2ce0bfef29 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:19.511858 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c687d43-5cf2-4f8a-9695-22baba464262 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:19.513022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c687d43-5cf2-4f8a-9695-22baba464262 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:19.513536 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c687d43-5cf2-4f8a-9695-22baba464262 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:19.514006 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c687d43-5cf2-4f8a-9695-22baba464262 None admin] PATH_INFO: `/v3/projects/2fc989279b744e4f93d785df902f0969` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:19.515390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c687d43-5cf2-4f8a-9695-22baba464262 None admin] RBAC: Authorizing `identity:get_project(project_id=2fc989279b744e4f93d785df902f0969)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:19.517135 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c687d43-5cf2-4f8a-9695-22baba464262 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:19.519401 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 578/1145] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:17:19 2022] GET /identity/v3/projects/2fc989279b744e4f93d785df902f0969 => generated 311 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:19.526347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e800646-9a61-4b4d-b017-7de950398b1e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:19.659732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e800646-9a61-4b4d-b017-7de950398b1e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:19.701583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230f1fa1-fab6-4412-b8bc-e0e5f879d2d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:19.702479 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230f1fa1-fab6-4412-b8bc-e0e5f879d2d9 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:19.702925 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230f1fa1-fab6-4412-b8bc-e0e5f879d2d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:19.703366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230f1fa1-fab6-4412-b8bc-e0e5f879d2d9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:19.704259 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230f1fa1-fab6-4412-b8bc-e0e5f879d2d9 None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=2fc989279b744e4f93d785df902f0969, user.id=d477c8a041ad4fa78c6f8aa50411137c) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:17:19.704748 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230f1fa1-fab6-4412-b8bc-e0e5f879d2d9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:19.706204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230f1fa1-fab6-4412-b8bc-e0e5f879d2d9 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:19.716356 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 568/1146] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:17:19 2022] GET /identity/v3/role_assignments?user.id=d477c8a041ad4fa78c6f8aa50411137c&scope.project.id=2fc989279b744e4f93d785df902f0969&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 259 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:17:21.942144 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2223a5-8c31-4e7a-9b6c-f0fcc2114c55 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:21.942803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2223a5-8c31-4e7a-9b6c-f0fcc2114c55 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:21.942803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2223a5-8c31-4e7a-9b6c-f0fcc2114c55 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:21.944155 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 579/1147] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:17:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:17:21.951282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de84d4c9-b4c8-4bdb-aeb0-ea7767c143d5 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:21.951826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de84d4c9-b4c8-4bdb-aeb0-ea7767c143d5 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:21.952289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de84d4c9-b4c8-4bdb-aeb0-ea7767c143d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:21.983368 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-de84d4c9-b4c8-4bdb-aeb0-ea7767c143d5 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:17:21.985126 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-de84d4c9-b4c8-4bdb-aeb0-ea7767c143d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:17:22.013227 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:22.090733 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 569/1148] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:17:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:17:22.360097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2223a5-8c31-4e7a-9b6c-f0fcc2114c55 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:22.470834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2223a5-8c31-4e7a-9b6c-f0fcc2114c55 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:22.508521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03341ce2-904c-4ce2-ab37-e4385ce52867 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:22.509343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03341ce2-904c-4ce2-ab37-e4385ce52867 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:22.509520 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03341ce2-904c-4ce2-ab37-e4385ce52867 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:22.509785 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03341ce2-904c-4ce2-ab37-e4385ce52867 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:22.522169 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03341ce2-904c-4ce2-ab37-e4385ce52867 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:17:22.527939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-03341ce2-904c-4ce2-ab37-e4385ce52867 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:17:22.527939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:17:22.527939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:17:22.527939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:17:22.527939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:17:22.527939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:17:22.527939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:17:22.527939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:17:22.527939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:17:22.527939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:17:22.527939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:17:22.527939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:17:22.527939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:17:22.527939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:17:22.527939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:17:22.527939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:17:22.529193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:17:22.529193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:17:22.529193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:17:22.529193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:17:22.529193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:17:22.529193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:17:22.529193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:17:22.529193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:17:22.529193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:17:22.529193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:17:22.529193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:17:22.529193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:17:22.529193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:17:22.529193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:17:22.529193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:17:22.529193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:17:22.530870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:17:22.530870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:17:22.530870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:17:22.530870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:17:22.530870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:17:22.530870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:17:22.530870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:17:22.530870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:17:22.530870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:17:22.530870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:17:22.530870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:17:22.530870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:17:22.530870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 580/1149] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:17:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:17:22.536536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de84d4c9-b4c8-4bdb-aeb0-ea7767c143d5 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:22.652907 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de84d4c9-b4c8-4bdb-aeb0-ea7767c143d5 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:22.688478 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a774688e-bb99-404a-9ca9-ad8322e5c12b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:22.689090 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a774688e-bb99-404a-9ca9-ad8322e5c12b None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:22.689370 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a774688e-bb99-404a-9ca9-ad8322e5c12b None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:22.689643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a774688e-bb99-404a-9ca9-ad8322e5c12b None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:22.690313 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a774688e-bb99-404a-9ca9-ad8322e5c12b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:17:22.690615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a774688e-bb99-404a-9ca9-ad8322e5c12b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:22.691995 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a774688e-bb99-404a-9ca9-ad8322e5c12b None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:22.702527 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 570/1150] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:17:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:22.709073 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03341ce2-904c-4ce2-ab37-e4385ce52867 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:22.741857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:22.824540 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03341ce2-904c-4ce2-ab37-e4385ce52867 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:22.863763 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb5207d-da5e-4df6-b8bb-c33106a2b221 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:22.864424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb5207d-da5e-4df6-b8bb-c33106a2b221 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:22.864698 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb5207d-da5e-4df6-b8bb-c33106a2b221 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:22.864965 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb5207d-da5e-4df6-b8bb-c33106a2b221 None admin] PATH_INFO: `/v3/users/d477c8a041ad4fa78c6f8aa50411137c` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:22.866097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb5207d-da5e-4df6-b8bb-c33106a2b221 None admin] RBAC: Authorizing `identity:get_user(user_id=d477c8a041ad4fa78c6f8aa50411137c)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:22.867572 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb5207d-da5e-4df6-b8bb-c33106a2b221 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:22.869424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 581/1151] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:17:22 2022] GET /identity/v3/users/d477c8a041ad4fa78c6f8aa50411137c => generated 286 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:22.875507 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a774688e-bb99-404a-9ca9-ad8322e5c12b None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:22.904454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:23.038261 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a774688e-bb99-404a-9ca9-ad8322e5c12b None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:23.095183 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e591f500-fe02-49f6-9ddc-21603d86dcec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:23.096262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e591f500-fe02-49f6-9ddc-21603d86dcec None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:23.096791 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e591f500-fe02-49f6-9ddc-21603d86dcec None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:23.097287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e591f500-fe02-49f6-9ddc-21603d86dcec None admin] PATH_INFO: `/v3/projects/2fc989279b744e4f93d785df902f0969` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:23.099370 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e591f500-fe02-49f6-9ddc-21603d86dcec None admin] RBAC: Authorizing `identity:get_project(project_id=2fc989279b744e4f93d785df902f0969)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:23.101840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e591f500-fe02-49f6-9ddc-21603d86dcec None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:23.105185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 571/1152] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:17:22 2022] GET /identity/v3/projects/2fc989279b744e4f93d785df902f0969 => generated 311 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:23.111572 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb5207d-da5e-4df6-b8bb-c33106a2b221 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:23.218368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:23.225665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb5207d-da5e-4df6-b8bb-c33106a2b221 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:23.265701 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71442d2e-d651-4ae4-aef5-d30dc253f4be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:23.266436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71442d2e-d651-4ae4-aef5-d30dc253f4be None admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:23.266755 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71442d2e-d651-4ae4-aef5-d30dc253f4be None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:23.267071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71442d2e-d651-4ae4-aef5-d30dc253f4be None admin] PATH_INFO: `/v3/projects/2fc989279b744e4f93d785df902f0969/users/d477c8a041ad4fa78c6f8aa50411137c/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:23.269217 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71442d2e-d651-4ae4-aef5-d30dc253f4be None admin] RBAC: Authorizing `identity:create_grant(project_id=2fc989279b744e4f93d785df902f0969, user_id=d477c8a041ad4fa78c6f8aa50411137c, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:23.270719 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71442d2e-d651-4ae4-aef5-d30dc253f4be None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:23.287219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 582/1153] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:17:23 2022] PUT /identity/v3/projects/2fc989279b744e4f93d785df902f0969/users/d477c8a041ad4fa78c6f8aa50411137c/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:17:25.567283 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eb5abd-dac7-4099-adf0-171be7eb7762 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:25.567844 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eb5abd-dac7-4099-adf0-171be7eb7762 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:25.567844 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eb5abd-dac7-4099-adf0-171be7eb7762 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:25.569346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 572/1154] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:17:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:17:25.576439 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f731a6fd-149c-4229-9034-758356edcc6d None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:25.577271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f731a6fd-149c-4229-9034-758356edcc6d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:25.577271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f731a6fd-149c-4229-9034-758356edcc6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:25.597033 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-f731a6fd-149c-4229-9034-758356edcc6d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:17:25.598288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f731a6fd-149c-4229-9034-758356edcc6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:17:25.674989 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 583/1155] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:17:25 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:17:25.956259 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19eb5abd-dac7-4099-adf0-171be7eb7762 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:26.055494 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:26.084330 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19eb5abd-dac7-4099-adf0-171be7eb7762 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:26.141021 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd3ec24-84af-44e0-b8e9-0a6b45dcf804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:26.142096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd3ec24-84af-44e0-b8e9-0a6b45dcf804 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:26.142561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd3ec24-84af-44e0-b8e9-0a6b45dcf804 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:26.142971 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd3ec24-84af-44e0-b8e9-0a6b45dcf804 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:26.162647 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd3ec24-84af-44e0-b8e9-0a6b45dcf804 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:17:26.172016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-7dd3ec24-84af-44e0-b8e9-0a6b45dcf804 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:17:26.172016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:17:26.172016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:17:26.172016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:17:26.172016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:17:26.172016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:17:26.172016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:17:26.172016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:17:26.172016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:17:26.172016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:17:26.172016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:17:26.172016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:17:26.172016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:17:26.172016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:17:26.172016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:17:26.172016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:17:26.173188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:17:26.173188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:17:26.173188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:17:26.173188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:17:26.173188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:17:26.173188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:17:26.173188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:17:26.173188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:17:26.173188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:17:26.173188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:17:26.173188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:17:26.173188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:17:26.173188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:17:26.173188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:17:26.173188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:17:26.173188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:17:26.174030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:17:26.174030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:17:26.174030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:17:26.174030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:17:26.174030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:17:26.174030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:17:26.174030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:17:26.174030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:17:26.174030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:17:26.174030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:17:26.174030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:17:26.174030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:17:26.174674 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 573/1156] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:17:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:17:26.181487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f731a6fd-149c-4229-9034-758356edcc6d None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:26.276236 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:26.295604 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f731a6fd-149c-4229-9034-758356edcc6d None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:26.332232 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a48253a-2824-4b23-88ab-23df1520a4ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:26.332912 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a48253a-2824-4b23-88ab-23df1520a4ab None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:26.333223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a48253a-2824-4b23-88ab-23df1520a4ab None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:26.333529 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a48253a-2824-4b23-88ab-23df1520a4ab None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:26.334259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a48253a-2824-4b23-88ab-23df1520a4ab None admin] RBAC: Adding query filter params (name=admin) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:17:26.334594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a48253a-2824-4b23-88ab-23df1520a4ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:26.336005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a48253a-2824-4b23-88ab-23df1520a4ab None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:26.345620 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 584/1157] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:17:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:17:26.351884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd3ec24-84af-44e0-b8e9-0a6b45dcf804 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:26.480044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd3ec24-84af-44e0-b8e9-0a6b45dcf804 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:26.517731 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999062cf-81f4-46fe-865d-fd2d4d3603c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:26.518371 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999062cf-81f4-46fe-865d-fd2d4d3603c6 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:26.518654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999062cf-81f4-46fe-865d-fd2d4d3603c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:26.518932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999062cf-81f4-46fe-865d-fd2d4d3603c6 None admin] PATH_INFO: `/v3/users/d477c8a041ad4fa78c6f8aa50411137c` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:26.520111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999062cf-81f4-46fe-865d-fd2d4d3603c6 None admin] RBAC: Authorizing `identity:get_user(user_id=d477c8a041ad4fa78c6f8aa50411137c)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:26.521588 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999062cf-81f4-46fe-865d-fd2d4d3603c6 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:26.523225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 574/1158] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:17:26 2022] GET /identity/v3/users/d477c8a041ad4fa78c6f8aa50411137c => generated 286 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:26.530955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a48253a-2824-4b23-88ab-23df1520a4ab None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:26.700039 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a48253a-2824-4b23-88ab-23df1520a4ab None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:26.737092 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b878e17-6b43-445e-ad7f-ece0e79a8649 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:26.737751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b878e17-6b43-445e-ad7f-ece0e79a8649 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:26.738071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b878e17-6b43-445e-ad7f-ece0e79a8649 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:26.738395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b878e17-6b43-445e-ad7f-ece0e79a8649 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:26.756527 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:26.842955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b878e17-6b43-445e-ad7f-ece0e79a8649 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:26.844377 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b878e17-6b43-445e-ad7f-ece0e79a8649 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:26.955873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 585/1159] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:17:26 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 14:17:26.963145 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999062cf-81f4-46fe-865d-fd2d4d3603c6 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:26.982180 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:27.082416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999062cf-81f4-46fe-865d-fd2d4d3603c6 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:27.123920 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7918704-3c19-45a4-8904-c88d68eceaec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:27.124655 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7918704-3c19-45a4-8904-c88d68eceaec None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:27.125000 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7918704-3c19-45a4-8904-c88d68eceaec None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:27.125323 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7918704-3c19-45a4-8904-c88d68eceaec None admin] PATH_INFO: `/v3/projects/2fc989279b744e4f93d785df902f0969` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:27.126602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7918704-3c19-45a4-8904-c88d68eceaec None admin] RBAC: Authorizing `identity:get_project(project_id=2fc989279b744e4f93d785df902f0969)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:27.128127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7918704-3c19-45a4-8904-c88d68eceaec None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:27.129919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 575/1160] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:17:26 2022] GET /identity/v3/projects/2fc989279b744e4f93d785df902f0969 => generated 311 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:27.136296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b878e17-6b43-445e-ad7f-ece0e79a8649 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:27.190541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:27.255473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b878e17-6b43-445e-ad7f-ece0e79a8649 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:27.296041 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90afd38-4951-4e38-9c14-f2271481d34c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:27.296681 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90afd38-4951-4e38-9c14-f2271481d34c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:27.296954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90afd38-4951-4e38-9c14-f2271481d34c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:27.297224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90afd38-4951-4e38-9c14-f2271481d34c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:27.297948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90afd38-4951-4e38-9c14-f2271481d34c None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=2fc989279b744e4f93d785df902f0969, user.id=d477c8a041ad4fa78c6f8aa50411137c) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:17:27.298252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90afd38-4951-4e38-9c14-f2271481d34c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:27.299684 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90afd38-4951-4e38-9c14-f2271481d34c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:27.306857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 586/1161] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:17:27 2022] GET /identity/v3/role_assignments?user.id=d477c8a041ad4fa78c6f8aa50411137c&scope.project.id=2fc989279b744e4f93d785df902f0969&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:29.624723 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99234cd-c4d1-486e-8090-254a01b4e118 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:29.625542 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99234cd-c4d1-486e-8090-254a01b4e118 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:29.625542 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99234cd-c4d1-486e-8090-254a01b4e118 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:29.626819 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 576/1162] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:17:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:17:29.632547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8d6048-9058-47d1-895e-b496f5411ae2 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:29.633328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8d6048-9058-47d1-895e-b496f5411ae2 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:29.633526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8d6048-9058-47d1-895e-b496f5411ae2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:29.654339 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-5a8d6048-9058-47d1-895e-b496f5411ae2 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:17:29.655623 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5a8d6048-9058-47d1-895e-b496f5411ae2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:17:29.732067 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 587/1163] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:17:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:17:30.006590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99234cd-c4d1-486e-8090-254a01b4e118 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:30.132836 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:30.171359 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99234cd-c4d1-486e-8090-254a01b4e118 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:30.228031 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f82d757-10e9-448e-8229-701d1fc90b2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:30.229262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f82d757-10e9-448e-8229-701d1fc90b2f None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:30.229769 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f82d757-10e9-448e-8229-701d1fc90b2f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:30.230273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f82d757-10e9-448e-8229-701d1fc90b2f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:30.354498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f82d757-10e9-448e-8229-701d1fc90b2f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:30.357138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f82d757-10e9-448e-8229-701d1fc90b2f None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:30.453753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:30.476911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 577/1164] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:17:30 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 471 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 14:17:30.483359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8d6048-9058-47d1-895e-b496f5411ae2 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:30.530733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:30.611160 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8d6048-9058-47d1-895e-b496f5411ae2 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:30.651680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1aa3d5-c947-46f7-bbe1-81d96d246580 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:30.652725 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1aa3d5-c947-46f7-bbe1-81d96d246580 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:30.652888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1aa3d5-c947-46f7-bbe1-81d96d246580 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:30.653146 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1aa3d5-c947-46f7-bbe1-81d96d246580 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:30.661521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1aa3d5-c947-46f7-bbe1-81d96d246580 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:30.662974 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1aa3d5-c947-46f7-bbe1-81d96d246580 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:30.676159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-ec1aa3d5-c947-46f7-bbe1-81d96d246580 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 04 14:17:30.676159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:17:30.676159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Oct 04 14:17:30.676159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Oct 04 14:17:30.676159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:17:30.676159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:17:30.676159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:17:30.676159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:17:30.676159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Oct 04 14:17:30.676159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:17:30.676159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:17:30.676159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:17:30.676159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:17:30.676159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:17:30.676159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:17:30.676159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:17:30.676159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:17:30.678143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:17:30.678143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:17:30.678143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:17:30.678143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:17:30.678143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:17:30.678143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:17:30.678143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:17:30.678143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:17:30.678143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:17:30.678143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:17:30.678143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:17:30.678143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:17:30.678143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:17:30.678143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:17:30.678143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:17:30.678143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:17:30.679807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:17:30.679807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:17:30.679807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:17:30.679807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:17:30.679807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:17:30.679807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:17:30.679807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:17:30.679807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:17:30.679807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:17:30.679807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:17:30.679807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Oct 04 14:17:30.679807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:17:30.679807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 04 14:17:30.679807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:17:30.679807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 588/1165] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Oct 4 14:17:30 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:17:30.684783 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f82d757-10e9-448e-8229-701d1fc90b2f None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:30.858003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f82d757-10e9-448e-8229-701d1fc90b2f None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:30.919321 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459f4036-b291-40a9-aa67-1715f90d11c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:30.920380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459f4036-b291-40a9-aa67-1715f90d11c3 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:30.920857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459f4036-b291-40a9-aa67-1715f90d11c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:30.921280 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459f4036-b291-40a9-aa67-1715f90d11c3 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:30.922411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459f4036-b291-40a9-aa67-1715f90d11c3 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:17:30.922950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459f4036-b291-40a9-aa67-1715f90d11c3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:30.925467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459f4036-b291-40a9-aa67-1715f90d11c3 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:30.939342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 578/1166] 10.222.0.64 () {56 vars in 1115 bytes} [Tue Oct 4 14:17:30 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:33.244997 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d796e1-6288-44ba-9c71-62dfd16d3245 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:33.245798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d796e1-6288-44ba-9c71-62dfd16d3245 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:33.245798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d796e1-6288-44ba-9c71-62dfd16d3245 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:33.246884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 589/1167] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:17:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:17:33.253696 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76badf1d-0056-41f8-8d28-52a44a44c602 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:33.254257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76badf1d-0056-41f8-8d28-52a44a44c602 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:33.254672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76badf1d-0056-41f8-8d28-52a44a44c602 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:33.284760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-76badf1d-0056-41f8-8d28-52a44a44c602 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:17:33.286977 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-76badf1d-0056-41f8-8d28-52a44a44c602 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:17:33.363502 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:33.409020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 579/1168] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:17:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:17:33.685952 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d796e1-6288-44ba-9c71-62dfd16d3245 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:33.800043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:33.804651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d796e1-6288-44ba-9c71-62dfd16d3245 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:33.846586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38405f82-3b6c-4c6c-9f49-7d21ac8e0aff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:33.847205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38405f82-3b6c-4c6c-9f49-7d21ac8e0aff None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:33.847620 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38405f82-3b6c-4c6c-9f49-7d21ac8e0aff None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:33.847947 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38405f82-3b6c-4c6c-9f49-7d21ac8e0aff None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:33.848890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38405f82-3b6c-4c6c-9f49-7d21ac8e0aff None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:33.850586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38405f82-3b6c-4c6c-9f49-7d21ac8e0aff None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:33.871572 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 590/1169] 10.222.0.64 () {60 vars in 1138 bytes} [Tue Oct 4 14:17:33 2022] POST /identity/v3/domains => generated 267 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:17:36.179539 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f3dc3b-d4a8-41af-a56f-e93f22262bb4 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:36.180655 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f3dc3b-d4a8-41af-a56f-e93f22262bb4 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:36.181024 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f3dc3b-d4a8-41af-a56f-e93f22262bb4 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:36.184738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 580/1170] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:17:36 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:17:36.193894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad88eb9-6ad9-4078-be6d-b8ae0b58fa18 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:36.194422 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad88eb9-6ad9-4078-be6d-b8ae0b58fa18 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:36.194835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad88eb9-6ad9-4078-be6d-b8ae0b58fa18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:36.224769 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-0ad88eb9-6ad9-4078-be6d-b8ae0b58fa18 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:17:36.226801 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-0ad88eb9-6ad9-4078-be6d-b8ae0b58fa18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:17:36.345523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 591/1171] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:17:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:17:36.624293 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f3dc3b-d4a8-41af-a56f-e93f22262bb4 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:36.747059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f3dc3b-d4a8-41af-a56f-e93f22262bb4 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:36.789275 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb5360d-9968-4790-8d00-80db4afc0337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:36.790175 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb5360d-9968-4790-8d00-80db4afc0337 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:36.790661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb5360d-9968-4790-8d00-80db4afc0337 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:36.790952 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb5360d-9968-4790-8d00-80db4afc0337 None admin] PATH_INFO: `/v3/domains/1fb051c8086549b691952ab304731e8e` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:36.804010 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb5360d-9968-4790-8d00-80db4afc0337 None admin] RBAC: Authorizing `identity:get_domain(domain_id=1fb051c8086549b691952ab304731e8e)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:36.806571 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb5360d-9968-4790-8d00-80db4afc0337 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:36.808727 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 581/1172] 10.222.0.64 () {56 vars in 1183 bytes} [Tue Oct 4 14:17:36 2022] GET /identity/v3/domains/1fb051c8086549b691952ab304731e8e => generated 267 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:17:36.815152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad88eb9-6ad9-4078-be6d-b8ae0b58fa18 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:36.929686 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:36.943963 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad88eb9-6ad9-4078-be6d-b8ae0b58fa18 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:36.984784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed19ee8-4e4b-48f2-b7c5-84a794c77c1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:36.985405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed19ee8-4e4b-48f2-b7c5-84a794c77c1a None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:36.985679 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed19ee8-4e4b-48f2-b7c5-84a794c77c1a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:36.985947 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed19ee8-4e4b-48f2-b7c5-84a794c77c1a None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:36.986715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed19ee8-4e4b-48f2-b7c5-84a794c77c1a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:36.988170 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed19ee8-4e4b-48f2-b7c5-84a794c77c1a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:37.018966 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 592/1173] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Oct 4 14:17:36 2022] POST /identity/v3/projects => generated 361 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:17:39.493724 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750adb77-7115-4c81-b481-32e0e9c4d340 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:39.494480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750adb77-7115-4c81-b481-32e0e9c4d340 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:39.494480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750adb77-7115-4c81-b481-32e0e9c4d340 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:39.495653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 582/1174] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:17:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:17:39.501995 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c26a05f-55d8-4b6e-8406-8871eceec713 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:39.502564 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c26a05f-55d8-4b6e-8406-8871eceec713 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:39.502564 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c26a05f-55d8-4b6e-8406-8871eceec713 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:39.524774 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-9c26a05f-55d8-4b6e-8406-8871eceec713 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:17:39.526764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9c26a05f-55d8-4b6e-8406-8871eceec713 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:17:39.618534 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 593/1175] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:17:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:17:39.898509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750adb77-7115-4c81-b481-32e0e9c4d340 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:39.938479 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:40.016725 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750adb77-7115-4c81-b481-32e0e9c4d340 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:40.053653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5fcab5-1532-457c-acb4-984869193f38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:40.054351 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5fcab5-1532-457c-acb4-984869193f38 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:40.054674 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5fcab5-1532-457c-acb4-984869193f38 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:40.055621 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5fcab5-1532-457c-acb4-984869193f38 None admin] PATH_INFO: `/v3/domains/1fb051c8086549b691952ab304731e8e` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:40.057133 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5fcab5-1532-457c-acb4-984869193f38 None admin] RBAC: Authorizing `identity:get_domain(domain_id=1fb051c8086549b691952ab304731e8e)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:40.058645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5fcab5-1532-457c-acb4-984869193f38 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:40.060371 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 583/1176] 10.222.0.64 () {56 vars in 1183 bytes} [Tue Oct 4 14:17:39 2022] GET /identity/v3/domains/1fb051c8086549b691952ab304731e8e => generated 267 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:40.066768 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c26a05f-55d8-4b6e-8406-8871eceec713 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:40.146083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:40.181287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c26a05f-55d8-4b6e-8406-8871eceec713 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:40.221259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccdaf1d-9cf9-4916-b9e8-3e0041157620 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:40.221874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccdaf1d-9cf9-4916-b9e8-3e0041157620 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:40.222151 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccdaf1d-9cf9-4916-b9e8-3e0041157620 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:40.222422 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccdaf1d-9cf9-4916-b9e8-3e0041157620 None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:40.223085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccdaf1d-9cf9-4916-b9e8-3e0041157620 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:40.224558 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccdaf1d-9cf9-4916-b9e8-3e0041157620 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:40.250665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 594/1177] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:17:40 2022] POST /identity/v3/users => generated 311 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:17:42.608890 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92479010-5180-4555-912a-4387839fc7ca None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:42.609546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92479010-5180-4555-912a-4387839fc7ca None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:42.609546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92479010-5180-4555-912a-4387839fc7ca None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:42.610978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 584/1178] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:17:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:17:42.617409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d54ad1c-46a8-41b0-8210-05e0245ff9c7 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:42.618127 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d54ad1c-46a8-41b0-8210-05e0245ff9c7 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:42.618127 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d54ad1c-46a8-41b0-8210-05e0245ff9c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:42.638088 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-6d54ad1c-46a8-41b0-8210-05e0245ff9c7 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:17:42.639453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6d54ad1c-46a8-41b0-8210-05e0245ff9c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:17:42.717020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 595/1179] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:17:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:17:42.990406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92479010-5180-4555-912a-4387839fc7ca None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:43.107063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92479010-5180-4555-912a-4387839fc7ca None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:43.109027 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:43.147294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9668dca9-c7d5-4be4-93bb-67f7ab4606cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:43.147935 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9668dca9-c7d5-4be4-93bb-67f7ab4606cf None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:43.148211 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9668dca9-c7d5-4be4-93bb-67f7ab4606cf None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:43.148512 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9668dca9-c7d5-4be4-93bb-67f7ab4606cf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:43.162178 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9668dca9-c7d5-4be4-93bb-67f7ab4606cf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:17:43.168122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-9668dca9-c7d5-4be4-93bb-67f7ab4606cf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:17:43.168122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:17:43.168122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:17:43.168122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:17:43.168122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:17:43.168122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:17:43.168122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:17:43.168122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:17:43.168122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:17:43.168122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:17:43.168122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:17:43.168122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:17:43.168122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:17:43.168122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:17:43.168122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:17:43.168122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:17:43.170094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:17:43.170094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:17:43.170094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:17:43.170094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:17:43.170094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:17:43.170094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:17:43.170094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:17:43.170094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:17:43.170094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:17:43.170094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:17:43.170094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:17:43.170094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:17:43.170094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:17:43.170094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:17:43.170094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:17:43.170094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:17:43.171759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:17:43.171759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:17:43.171759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:17:43.171759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:17:43.171759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:17:43.171759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:17:43.171759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:17:43.171759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:17:43.171759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:17:43.171759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:17:43.171759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:17:43.171759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:17:43.171759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 585/1180] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:17:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:17:43.179289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d54ad1c-46a8-41b0-8210-05e0245ff9c7 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:43.250540 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:43.298682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d54ad1c-46a8-41b0-8210-05e0245ff9c7 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:43.339719 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d4abfd-fdc7-4a14-8136-62f477958ef1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:43.340351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d4abfd-fdc7-4a14-8136-62f477958ef1 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:43.340628 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d4abfd-fdc7-4a14-8136-62f477958ef1 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:43.340896 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d4abfd-fdc7-4a14-8136-62f477958ef1 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:43.341563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d4abfd-fdc7-4a14-8136-62f477958ef1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:17:43.341859 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d4abfd-fdc7-4a14-8136-62f477958ef1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:43.343240 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d4abfd-fdc7-4a14-8136-62f477958ef1 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:43.353307 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 596/1181] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:17:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:43.359838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9668dca9-c7d5-4be4-93bb-67f7ab4606cf None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:43.486818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9668dca9-c7d5-4be4-93bb-67f7ab4606cf None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:43.524630 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d47f47-8054-49a9-87bb-eccc37a5ff3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:43.525333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d47f47-8054-49a9-87bb-eccc37a5ff3b None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:43.525662 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d47f47-8054-49a9-87bb-eccc37a5ff3b None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:43.525981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d47f47-8054-49a9-87bb-eccc37a5ff3b None admin] PATH_INFO: `/v3/users/311f924d3f994c9998a8acad5e42ff72` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:43.542113 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d47f47-8054-49a9-87bb-eccc37a5ff3b None admin] RBAC: Authorizing `identity:get_user(user_id=311f924d3f994c9998a8acad5e42ff72)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:43.543696 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d47f47-8054-49a9-87bb-eccc37a5ff3b None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:43.545578 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 586/1182] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:17:43 2022] GET /identity/v3/users/311f924d3f994c9998a8acad5e42ff72 => generated 311 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:43.551610 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d4abfd-fdc7-4a14-8136-62f477958ef1 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:43.665560 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d4abfd-fdc7-4a14-8136-62f477958ef1 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:43.688356 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:43.705716 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0001370-5e00-4e98-bcfc-5e8c81b823b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:43.706397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0001370-5e00-4e98-bcfc-5e8c81b823b6 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:43.706713 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0001370-5e00-4e98-bcfc-5e8c81b823b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:43.707020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0001370-5e00-4e98-bcfc-5e8c81b823b6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:43.782281 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0001370-5e00-4e98-bcfc-5e8c81b823b6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:43.783661 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0001370-5e00-4e98-bcfc-5e8c81b823b6 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:43.901511 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 597/1183] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:17:43 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 14:17:43.908521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d47f47-8054-49a9-87bb-eccc37a5ff3b None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:43.933288 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:44.074653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d47f47-8054-49a9-87bb-eccc37a5ff3b None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:44.135592 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da067b50-65b9-438a-92ce-c77aa62264bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:44.136694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da067b50-65b9-438a-92ce-c77aa62264bc None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:44.137194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da067b50-65b9-438a-92ce-c77aa62264bc None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:44.137690 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da067b50-65b9-438a-92ce-c77aa62264bc None admin] PATH_INFO: `/v3/projects/6bb85722e55e440f982c16e4a001c44f` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:44.139643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da067b50-65b9-438a-92ce-c77aa62264bc None admin] RBAC: Authorizing `identity:get_project(project_id=6bb85722e55e440f982c16e4a001c44f)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:44.142099 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da067b50-65b9-438a-92ce-c77aa62264bc None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:44.145209 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 587/1184] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:17:43 2022] GET /identity/v3/projects/6bb85722e55e440f982c16e4a001c44f => generated 361 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:44.152183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0001370-5e00-4e98-bcfc-5e8c81b823b6 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:44.174937 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:44.266494 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0001370-5e00-4e98-bcfc-5e8c81b823b6 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:44.306210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1883d77-a2f9-4efe-b891-ac9433431aa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:44.306837 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1883d77-a2f9-4efe-b891-ac9433431aa4 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:44.307109 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1883d77-a2f9-4efe-b891-ac9433431aa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:44.307373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1883d77-a2f9-4efe-b891-ac9433431aa4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:44.308129 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1883d77-a2f9-4efe-b891-ac9433431aa4 None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=6bb85722e55e440f982c16e4a001c44f, user.id=311f924d3f994c9998a8acad5e42ff72) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:17:44.308466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1883d77-a2f9-4efe-b891-ac9433431aa4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:44.309941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1883d77-a2f9-4efe-b891-ac9433431aa4 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:44.316447 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 598/1185] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:17:44 2022] GET /identity/v3/role_assignments?user.id=311f924d3f994c9998a8acad5e42ff72&scope.project.id=6bb85722e55e440f982c16e4a001c44f&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 259 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:46.573566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00f0340-8bd7-4de8-b0e5-a7e9d5804e3f None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:46.574136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00f0340-8bd7-4de8-b0e5-a7e9d5804e3f None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:46.574136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00f0340-8bd7-4de8-b0e5-a7e9d5804e3f None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:46.575509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 588/1186] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:17:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:17:46.581496 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a44b39-a85d-4960-a2be-53272f05b950 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:46.582016 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a44b39-a85d-4960-a2be-53272f05b950 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:46.582016 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a44b39-a85d-4960-a2be-53272f05b950 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:46.602838 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-b8a44b39-a85d-4960-a2be-53272f05b950 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:17:46.604168 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b8a44b39-a85d-4960-a2be-53272f05b950 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:17:46.677649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 599/1187] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:17:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:17:46.955617 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00f0340-8bd7-4de8-b0e5-a7e9d5804e3f None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:47.056879 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:47.072873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00f0340-8bd7-4de8-b0e5-a7e9d5804e3f None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:47.111244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427203b6-a19a-4b3d-a55c-156ee37c93e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:47.111894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427203b6-a19a-4b3d-a55c-156ee37c93e1 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:47.112217 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427203b6-a19a-4b3d-a55c-156ee37c93e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:47.112514 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427203b6-a19a-4b3d-a55c-156ee37c93e1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:47.125525 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427203b6-a19a-4b3d-a55c-156ee37c93e1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:17:47.130521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-427203b6-a19a-4b3d-a55c-156ee37c93e1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:17:47.130521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:17:47.130521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:17:47.130521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:17:47.130521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:17:47.130521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:17:47.130521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:17:47.130521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:17:47.130521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:17:47.130521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:17:47.130521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:17:47.130521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:17:47.130521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:17:47.130521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:17:47.130521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:17:47.130521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:17:47.132265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:17:47.132265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:17:47.132265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:17:47.132265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:17:47.132265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:17:47.132265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:17:47.132265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:17:47.132265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:17:47.132265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:17:47.132265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:17:47.132265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:17:47.132265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:17:47.132265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:17:47.132265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:17:47.132265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:17:47.132265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:17:47.134581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:17:47.134581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:17:47.134581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:17:47.134581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:17:47.134581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:17:47.134581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:17:47.134581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:17:47.134581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:17:47.134581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:17:47.134581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:17:47.134581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:17:47.134581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:17:47.134581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 589/1188] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:17:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:17:47.140774 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a44b39-a85d-4960-a2be-53272f05b950 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:47.157399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:47.256617 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a44b39-a85d-4960-a2be-53272f05b950 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:47.296136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d071e2-e4c9-4cdd-8847-213c5321a910 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:47.296819 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d071e2-e4c9-4cdd-8847-213c5321a910 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:47.297131 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d071e2-e4c9-4cdd-8847-213c5321a910 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:47.297440 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d071e2-e4c9-4cdd-8847-213c5321a910 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:47.298179 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d071e2-e4c9-4cdd-8847-213c5321a910 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:17:47.298518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d071e2-e4c9-4cdd-8847-213c5321a910 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:47.299931 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d071e2-e4c9-4cdd-8847-213c5321a910 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:47.310586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 600/1189] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:17:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:47.316735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427203b6-a19a-4b3d-a55c-156ee37c93e1 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:47.446451 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427203b6-a19a-4b3d-a55c-156ee37c93e1 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:47.489809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1462e43-af04-4f58-9224-50b67db46159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:47.490443 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1462e43-af04-4f58-9224-50b67db46159 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:47.490717 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1462e43-af04-4f58-9224-50b67db46159 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:47.490987 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1462e43-af04-4f58-9224-50b67db46159 None admin] PATH_INFO: `/v3/users/311f924d3f994c9998a8acad5e42ff72` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:47.492380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1462e43-af04-4f58-9224-50b67db46159 None admin] RBAC: Authorizing `identity:get_user(user_id=311f924d3f994c9998a8acad5e42ff72)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:47.493924 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1462e43-af04-4f58-9224-50b67db46159 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:47.495661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 590/1190] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:17:47 2022] GET /identity/v3/users/311f924d3f994c9998a8acad5e42ff72 => generated 311 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:17:47.502251 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d071e2-e4c9-4cdd-8847-213c5321a910 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:47.590325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:47.616834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d071e2-e4c9-4cdd-8847-213c5321a910 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:47.661268 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd614e6-2d57-4775-a915-5906f1f5a432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:47.661907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd614e6-2d57-4775-a915-5906f1f5a432 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:47.662178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd614e6-2d57-4775-a915-5906f1f5a432 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:47.662446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd614e6-2d57-4775-a915-5906f1f5a432 None admin] PATH_INFO: `/v3/projects/6bb85722e55e440f982c16e4a001c44f` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:47.663625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd614e6-2d57-4775-a915-5906f1f5a432 None admin] RBAC: Authorizing `identity:get_project(project_id=6bb85722e55e440f982c16e4a001c44f)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:47.665158 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd614e6-2d57-4775-a915-5906f1f5a432 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:47.667348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 601/1191] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:17:47 2022] GET /identity/v3/projects/6bb85722e55e440f982c16e4a001c44f => generated 361 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:47.673902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1462e43-af04-4f58-9224-50b67db46159 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:47.694163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:47.808332 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1462e43-af04-4f58-9224-50b67db46159 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:47.847500 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8630306f-8f3c-4952-879c-993ac5222071 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:47.848136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8630306f-8f3c-4952-879c-993ac5222071 None admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:47.848492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8630306f-8f3c-4952-879c-993ac5222071 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:47.848840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8630306f-8f3c-4952-879c-993ac5222071 None admin] PATH_INFO: `/v3/projects/6bb85722e55e440f982c16e4a001c44f/users/311f924d3f994c9998a8acad5e42ff72/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:47.851216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8630306f-8f3c-4952-879c-993ac5222071 None admin] RBAC: Authorizing `identity:create_grant(project_id=6bb85722e55e440f982c16e4a001c44f, user_id=311f924d3f994c9998a8acad5e42ff72, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:47.853409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8630306f-8f3c-4952-879c-993ac5222071 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:47.867892 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 591/1192] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:17:47 2022] PUT /identity/v3/projects/6bb85722e55e440f982c16e4a001c44f/users/311f924d3f994c9998a8acad5e42ff72/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:17:50.116567 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b57244-e98e-4f0d-aaa5-897ef936ea46 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:50.117736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b57244-e98e-4f0d-aaa5-897ef936ea46 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:50.117736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b57244-e98e-4f0d-aaa5-897ef936ea46 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:50.119277 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 602/1193] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:17:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:17:50.125759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c910f96f-21c4-4ca4-b1af-90cad75ea5fe None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:50.126336 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c910f96f-21c4-4ca4-b1af-90cad75ea5fe None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:50.126336 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c910f96f-21c4-4ca4-b1af-90cad75ea5fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:50.147840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-c910f96f-21c4-4ca4-b1af-90cad75ea5fe None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:17:50.149830 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c910f96f-21c4-4ca4-b1af-90cad75ea5fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:17:50.235982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 592/1194] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:17:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:17:50.511584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b57244-e98e-4f0d-aaa5-897ef936ea46 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:50.623649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b57244-e98e-4f0d-aaa5-897ef936ea46 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:50.659819 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7996a11e-cf00-4c7c-8906-7cfdd2c8ae8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:50.660480 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7996a11e-cf00-4c7c-8906-7cfdd2c8ae8f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:50.660758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7996a11e-cf00-4c7c-8906-7cfdd2c8ae8f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:50.661028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7996a11e-cf00-4c7c-8906-7cfdd2c8ae8f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:50.673241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7996a11e-cf00-4c7c-8906-7cfdd2c8ae8f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:17:50.678101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-7996a11e-cf00-4c7c-8906-7cfdd2c8ae8f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:17:50.678101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:17:50.678101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:17:50.678101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:17:50.678101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:17:50.678101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:17:50.678101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:17:50.678101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:17:50.678101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:17:50.678101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:17:50.678101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:17:50.678101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:17:50.678101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:17:50.678101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:17:50.678101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:17:50.678101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:17:50.679469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:17:50.679469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:17:50.679469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:17:50.679469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:17:50.679469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:17:50.679469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:17:50.679469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:17:50.679469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:17:50.679469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:17:50.679469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:17:50.679469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:17:50.679469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:17:50.679469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:17:50.679469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:17:50.679469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:17:50.679469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:17:50.681300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:17:50.681300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:17:50.681300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:17:50.681300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:17:50.681300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:17:50.681300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:17:50.681300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:17:50.681300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:17:50.681300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:17:50.681300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:17:50.681300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:17:50.681300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:17:50.684496 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 603/1195] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:17:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:17:50.686392 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c910f96f-21c4-4ca4-b1af-90cad75ea5fe None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:50.695198 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:50.796884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c910f96f-21c4-4ca4-b1af-90cad75ea5fe None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:50.836634 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced14d0d-e7a2-4d67-abd6-1a2067cca5d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:50.837260 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced14d0d-e7a2-4d67-abd6-1a2067cca5d9 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:50.837543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced14d0d-e7a2-4d67-abd6-1a2067cca5d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:50.837818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced14d0d-e7a2-4d67-abd6-1a2067cca5d9 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:50.838495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced14d0d-e7a2-4d67-abd6-1a2067cca5d9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:17:50.838800 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced14d0d-e7a2-4d67-abd6-1a2067cca5d9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:50.840351 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced14d0d-e7a2-4d67-abd6-1a2067cca5d9 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:50.851128 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 593/1196] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:17:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:50.857897 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7996a11e-cf00-4c7c-8906-7cfdd2c8ae8f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:50.861904 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:50.966446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7996a11e-cf00-4c7c-8906-7cfdd2c8ae8f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:51.002943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d620a49a-265e-4421-928f-05c36ebbdf3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:51.003584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d620a49a-265e-4421-928f-05c36ebbdf3d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:51.003862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d620a49a-265e-4421-928f-05c36ebbdf3d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:51.004134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d620a49a-265e-4421-928f-05c36ebbdf3d None admin] PATH_INFO: `/v3/users/311f924d3f994c9998a8acad5e42ff72` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:51.005328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d620a49a-265e-4421-928f-05c36ebbdf3d None admin] RBAC: Authorizing `identity:get_user(user_id=311f924d3f994c9998a8acad5e42ff72)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:51.006769 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d620a49a-265e-4421-928f-05c36ebbdf3d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:51.009356 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 604/1197] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Oct 4 14:17:50 2022] GET /identity/v3/users/311f924d3f994c9998a8acad5e42ff72 => generated 311 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:17:51.016354 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced14d0d-e7a2-4d67-abd6-1a2067cca5d9 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:51.089333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:51.131672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced14d0d-e7a2-4d67-abd6-1a2067cca5d9 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:51.171938 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e841e1-7b8b-4421-9985-0f24ede09340 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:51.172597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e841e1-7b8b-4421-9985-0f24ede09340 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:51.172876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e841e1-7b8b-4421-9985-0f24ede09340 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:51.173149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e841e1-7b8b-4421-9985-0f24ede09340 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:51.253247 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e841e1-7b8b-4421-9985-0f24ede09340 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:51.254738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e841e1-7b8b-4421-9985-0f24ede09340 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:51.328658 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:51.370490 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 594/1198] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:17:51 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 14:17:51.376876 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d620a49a-265e-4421-928f-05c36ebbdf3d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:51.463302 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:17:51.500813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d620a49a-265e-4421-928f-05c36ebbdf3d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:51.541227 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b55a5d-e309-4d8d-91c6-5e343882d03c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:51.541934 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b55a5d-e309-4d8d-91c6-5e343882d03c None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:51.542258 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b55a5d-e309-4d8d-91c6-5e343882d03c None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:51.542575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b55a5d-e309-4d8d-91c6-5e343882d03c None admin] PATH_INFO: `/v3/projects/6bb85722e55e440f982c16e4a001c44f` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:51.543806 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b55a5d-e309-4d8d-91c6-5e343882d03c None admin] RBAC: Authorizing `identity:get_project(project_id=6bb85722e55e440f982c16e4a001c44f)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:51.545414 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b55a5d-e309-4d8d-91c6-5e343882d03c None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:51.547284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 605/1199] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Oct 4 14:17:51 2022] GET /identity/v3/projects/6bb85722e55e440f982c16e4a001c44f => generated 361 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:17:51.555483 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e841e1-7b8b-4421-9985-0f24ede09340 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:17:51.670396 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e841e1-7b8b-4421-9985-0f24ede09340 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:17:51.709404 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6b4ebd-7978-4e06-bc2b-a9ca3ebc186f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:17:51.710016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6b4ebd-7978-4e06-bc2b-a9ca3ebc186f None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:17:51.710299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6b4ebd-7978-4e06-bc2b-a9ca3ebc186f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:17:51.711118 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6b4ebd-7978-4e06-bc2b-a9ca3ebc186f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:17:51.712145 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6b4ebd-7978-4e06-bc2b-a9ca3ebc186f None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=6bb85722e55e440f982c16e4a001c44f, user.id=311f924d3f994c9998a8acad5e42ff72) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:17:51.712487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6b4ebd-7978-4e06-bc2b-a9ca3ebc186f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:17:51.713937 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6b4ebd-7978-4e06-bc2b-a9ca3ebc186f None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:17:51.720659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 595/1200] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:17:51 2022] GET /identity/v3/role_assignments?user.id=311f924d3f994c9998a8acad5e42ff72&scope.project.id=6bb85722e55e440f982c16e4a001c44f&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 608 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:18:01.535840 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=30 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:18:01.718103 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=84 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:18:54.578171 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a83b4d6-9958-4550-8f9a-a63257f92eb4 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:18:54.579243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a83b4d6-9958-4550-8f9a-a63257f92eb4 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:18:54.579243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a83b4d6-9958-4550-8f9a-a63257f92eb4 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:18:54.582221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 606/1201] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:18:54 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:18:54.592820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519c0b33-f5f0-44a5-a672-87680a49f970 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:18:54.593738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519c0b33-f5f0-44a5-a672-87680a49f970 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:18:54.593738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519c0b33-f5f0-44a5-a672-87680a49f970 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:18:54.625916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-519c0b33-f5f0-44a5-a672-87680a49f970 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:18:54.627897 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-519c0b33-f5f0-44a5-a672-87680a49f970 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:18:54.781105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 596/1202] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:18:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:18:55.324863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a83b4d6-9958-4550-8f9a-a63257f92eb4 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:18:55.579667 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a83b4d6-9958-4550-8f9a-a63257f92eb4 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:18:55.654032 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfedf401-370b-4c1d-b29b-a381f20d648a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:18:55.654799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfedf401-370b-4c1d-b29b-a381f20d648a None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:18:55.655064 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfedf401-370b-4c1d-b29b-a381f20d648a None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:18:55.655308 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfedf401-370b-4c1d-b29b-a381f20d648a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:18:55.669683 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfedf401-370b-4c1d-b29b-a381f20d648a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:18:55.671161 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfedf401-370b-4c1d-b29b-a381f20d648a None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:18:55.694166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-cfedf401-370b-4c1d-b29b-a381f20d648a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:18:55.694166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:18:55.694166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:18:55.694166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:18:55.694166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:18:55.694166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:18:55.694166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:18:55.694166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:18:55.694166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:18:55.694166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:18:55.694166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:18:55.694166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:18:55.694166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:18:55.694166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:18:55.694166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:18:55.694166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:18:55.696600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:18:55.696600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:18:55.696600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:18:55.696600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:18:55.696600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:18:55.696600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:18:55.696600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:18:55.696600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:18:55.696600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:18:55.696600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:18:55.696600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:18:55.696600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:18:55.696600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:18:55.696600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:18:55.696600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:18:55.696600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:18:55.696600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:18:55.698789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:18:55.698789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:18:55.698789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:18:55.698789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 607/1203] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:18:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 372 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:18:55.705322 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519c0b33-f5f0-44a5-a672-87680a49f970 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:18:55.938773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519c0b33-f5f0-44a5-a672-87680a49f970 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:18:55.996938 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:18:56.005292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c88476e-27b5-4118-baa7-f3f808dd378c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:18:56.006106 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c88476e-27b5-4118-baa7-f3f808dd378c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:18:56.006582 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c88476e-27b5-4118-baa7-f3f808dd378c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:18:56.006943 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c88476e-27b5-4118-baa7-f3f808dd378c None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:18:56.007760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c88476e-27b5-4118-baa7-f3f808dd378c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:18:56.008157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c88476e-27b5-4118-baa7-f3f808dd378c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:18:56.009785 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c88476e-27b5-4118-baa7-f3f808dd378c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:18:56.024552 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 597/1204] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:18:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:18:56.032086 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfedf401-370b-4c1d-b29b-a381f20d648a None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:18:56.134564 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:18:56.229849 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfedf401-370b-4c1d-b29b-a381f20d648a None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:18:56.303049 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbe708a-307d-4e61-85ac-b43265dca6cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:18:56.304118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbe708a-307d-4e61-85ac-b43265dca6cf None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:18:56.304701 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbe708a-307d-4e61-85ac-b43265dca6cf None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:18:56.305159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbe708a-307d-4e61-85ac-b43265dca6cf None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:18:56.306486 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbe708a-307d-4e61-85ac-b43265dca6cf None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:18:56.309270 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbe708a-307d-4e61-85ac-b43265dca6cf None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:18:56.358697 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 608/1205] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Oct 4 14:18:56 2022] POST /identity/v3/users => generated 251 bytes in 328 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:18:59.766873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe92665-4597-4417-ba63-57cd33eb5e22 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:18:59.767673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe92665-4597-4417-ba63-57cd33eb5e22 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:18:59.767673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe92665-4597-4417-ba63-57cd33eb5e22 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:18:59.768988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 598/1206] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:18:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:18:59.775877 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d1c7fc-f73e-4081-b7ee-0b2bbe0378db None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:18:59.776571 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d1c7fc-f73e-4081-b7ee-0b2bbe0378db None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:18:59.776571 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d1c7fc-f73e-4081-b7ee-0b2bbe0378db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:18:59.796911 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-b0d1c7fc-f73e-4081-b7ee-0b2bbe0378db None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:18:59.798152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b0d1c7fc-f73e-4081-b7ee-0b2bbe0378db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:18:59.895193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 609/1207] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:18:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:19:00.245029 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe92665-4597-4417-ba63-57cd33eb5e22 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:00.414367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe92665-4597-4417-ba63-57cd33eb5e22 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:00.471804 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8292d1d5-46ee-43d0-badb-374cbc70cb6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:00.472531 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8292d1d5-46ee-43d0-badb-374cbc70cb6d None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:00.473339 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8292d1d5-46ee-43d0-badb-374cbc70cb6d None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:00.473615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8292d1d5-46ee-43d0-badb-374cbc70cb6d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:00.486933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8292d1d5-46ee-43d0-badb-374cbc70cb6d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:19:00.493134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-8292d1d5-46ee-43d0-badb-374cbc70cb6d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:19:00.493134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:00.493134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:00.493134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:00.493134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:00.493134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:00.493134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:00.493134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:00.493134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:00.493134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:00.493134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:00.493134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:00.493134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:19:00.493134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:19:00.493134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:19:00.493134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:19:00.495906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:19:00.495906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:19:00.495906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:19:00.495906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:19:00.495906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:00.495906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:00.495906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:00.495906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:00.495906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:00.495906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:00.495906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:00.495906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:00.495906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:00.495906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:00.495906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:00.495906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:00.497931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:00.497931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:00.497931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:00.497931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:00.497931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:19:00.497931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:19:00.497931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:19:00.497931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:19:00.497931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:19:00.497931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:19:00.497931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:19:00.497931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:00.497931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 599/1208] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:19:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:00.500834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d1c7fc-f73e-4081-b7ee-0b2bbe0378db None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:00.574147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:00.713090 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d1c7fc-f73e-4081-b7ee-0b2bbe0378db None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:00.775908 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0085d031-8fd7-4470-b574-ba3ca046a33e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:00.777178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0085d031-8fd7-4470-b574-ba3ca046a33e None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:00.777330 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0085d031-8fd7-4470-b574-ba3ca046a33e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:00.777932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0085d031-8fd7-4470-b574-ba3ca046a33e None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:00.778762 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0085d031-8fd7-4470-b574-ba3ca046a33e None admin] RBAC: Adding query filter params (name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:00.779219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0085d031-8fd7-4470-b574-ba3ca046a33e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:00.781641 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0085d031-8fd7-4470-b574-ba3ca046a33e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:00.796113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 610/1209] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:19:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:19:00.804196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8292d1d5-46ee-43d0-badb-374cbc70cb6d None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:00.896368 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:00.991686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8292d1d5-46ee-43d0-badb-374cbc70cb6d None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:01.064460 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20e83e8-2d42-4d08-a9de-ba7d3f50e626 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:01.064855 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20e83e8-2d42-4d08-a9de-ba7d3f50e626 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:01.065207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20e83e8-2d42-4d08-a9de-ba7d3f50e626 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:01.065701 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20e83e8-2d42-4d08-a9de-ba7d3f50e626 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:01.087440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20e83e8-2d42-4d08-a9de-ba7d3f50e626 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:01.089289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20e83e8-2d42-4d08-a9de-ba7d3f50e626 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:01.111229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-c20e83e8-2d42-4d08-a9de-ba7d3f50e626 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:01.111229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:01.111229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:01.111229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:01.111229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:01.111229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:01.111229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:01.111229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:01.111229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:01.111229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:01.111229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:01.111229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:01.111229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:19:01.111229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:19:01.111229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:19:01.111229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:19:01.116414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:01.116414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:01.116414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:01.116414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:01.116414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:01.116414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:01.116414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:01.116414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:01.116414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:01.116414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:01.116414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:01.116414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:01.116414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:01.116414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:01.116414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:01.116414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:01.116414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:19:01.118333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:19:01.118333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:01.118333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:01.118333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 600/1210] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:19:01.121974 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0085d031-8fd7-4470-b574-ba3ca046a33e None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:01.277090 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:01.283685 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0085d031-8fd7-4470-b574-ba3ca046a33e None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:01.349902 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7c41cb-564d-40b1-b72d-ac26a94e9ff9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:01.350637 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7c41cb-564d-40b1-b72d-ac26a94e9ff9 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:01.350929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7c41cb-564d-40b1-b72d-ac26a94e9ff9 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:01.351229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7c41cb-564d-40b1-b72d-ac26a94e9ff9 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:01.351958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7c41cb-564d-40b1-b72d-ac26a94e9ff9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:01.352302 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7c41cb-564d-40b1-b72d-ac26a94e9ff9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:01.353920 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7c41cb-564d-40b1-b72d-ac26a94e9ff9 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:01.373457 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 611/1211] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:19:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:19:01.383656 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20e83e8-2d42-4d08-a9de-ba7d3f50e626 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:01.555837 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20e83e8-2d42-4d08-a9de-ba7d3f50e626 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:01.558684 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:01.612731 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683436ed-a35c-4909-8a4c-6abda6f7109e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:01.614651 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683436ed-a35c-4909-8a4c-6abda6f7109e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:01.615208 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683436ed-a35c-4909-8a4c-6abda6f7109e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:01.615709 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683436ed-a35c-4909-8a4c-6abda6f7109e None admin] PATH_INFO: `/v3/users/placement` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:01.634287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683436ed-a35c-4909-8a4c-6abda6f7109e None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:01.637603 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683436ed-a35c-4909-8a4c-6abda6f7109e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:01.666415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-683436ed-a35c-4909-8a4c-6abda6f7109e None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 14:19:01.666415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:01.666415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:01.666415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:01.666415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:01.666415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:01.666415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:01.666415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:01.666415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:01.666415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:01.666415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:01.666415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:01.666415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:19:01.666415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:19:01.666415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:19:01.666415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:01.668821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:01.671119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:01.671119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:01.671119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:19:01.671119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:19:01.671119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:19:01.671119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:19:01.671119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:19:01.671119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:19:01.671119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 14:19:01.671119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:01.671119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 601/1212] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:01 2022] GET /identity/v3/users/placement => generated 87 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:01.678289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7c41cb-564d-40b1-b72d-ac26a94e9ff9 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:01.887799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7c41cb-564d-40b1-b72d-ac26a94e9ff9 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:01.964484 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382f1b43-de81-4d04-9207-d2315785dd8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:01.965488 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382f1b43-de81-4d04-9207-d2315785dd8f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:01.965958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382f1b43-de81-4d04-9207-d2315785dd8f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:01.966431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382f1b43-de81-4d04-9207-d2315785dd8f None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:01.967831 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382f1b43-de81-4d04-9207-d2315785dd8f None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:01.968381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382f1b43-de81-4d04-9207-d2315785dd8f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:01.971154 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382f1b43-de81-4d04-9207-d2315785dd8f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:02.010074 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 612/1213] 10.222.0.64 () {56 vars in 1143 bytes} [Tue Oct 4 14:19:01 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 333 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:02.021293 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683436ed-a35c-4909-8a4c-6abda6f7109e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:02.219545 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683436ed-a35c-4909-8a4c-6abda6f7109e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:02.272437 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7894a6a-0c0f-47f8-abf1-f8abfb2413c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:02.273230 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7894a6a-0c0f-47f8-abf1-f8abfb2413c8 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:02.273501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7894a6a-0c0f-47f8-abf1-f8abfb2413c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:02.273752 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7894a6a-0c0f-47f8-abf1-f8abfb2413c8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:02.321515 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:02.383168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7894a6a-0c0f-47f8-abf1-f8abfb2413c8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:02.383168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7894a6a-0c0f-47f8-abf1-f8abfb2413c8 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:02.591019 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 602/1214] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:19:02 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 572 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 14:19:02.599685 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382f1b43-de81-4d04-9207-d2315785dd8f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:02.641224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:02.811039 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382f1b43-de81-4d04-9207-d2315785dd8f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:02.861792 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c78cccc-53c7-4997-a01c-649bc7bbaeef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:02.862373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c78cccc-53c7-4997-a01c-649bc7bbaeef None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:02.862726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c78cccc-53c7-4997-a01c-649bc7bbaeef None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:02.863743 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c78cccc-53c7-4997-a01c-649bc7bbaeef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:02.878376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c78cccc-53c7-4997-a01c-649bc7bbaeef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:02.880804 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c78cccc-53c7-4997-a01c-649bc7bbaeef None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:02.901756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-8c78cccc-53c7-4997-a01c-649bc7bbaeef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:02.901756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:02.901756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:02.901756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:02.901756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:02.901756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:02.901756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:02.901756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:02.901756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:02.901756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:02.901756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:02.901756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:02.901756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:19:02.901756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:19:02.901756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:19:02.901756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:19:02.904134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:02.904134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:02.904134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:02.904134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:02.904134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:02.904134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:02.904134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:02.904134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:02.904134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:02.904134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:02.904134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:02.904134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:02.904134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:02.904134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:02.904134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:02.904134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:02.904134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:19:02.905879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:19:02.905879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:02.905879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:19:02.905879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 613/1215] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:19:02.911999 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7894a6a-0c0f-47f8-abf1-f8abfb2413c8 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:03.038291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:03.104436 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7894a6a-0c0f-47f8-abf1-f8abfb2413c8 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:03.155305 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623cb60f-c6df-4afe-ab30-96a46076cfb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:03.156138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623cb60f-c6df-4afe-ab30-96a46076cfb5 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:03.156563 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623cb60f-c6df-4afe-ab30-96a46076cfb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:03.156991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623cb60f-c6df-4afe-ab30-96a46076cfb5 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:03.157820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623cb60f-c6df-4afe-ab30-96a46076cfb5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:03.158275 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623cb60f-c6df-4afe-ab30-96a46076cfb5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:03.159997 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623cb60f-c6df-4afe-ab30-96a46076cfb5 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:03.178427 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 603/1216] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:19:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:03.185295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c78cccc-53c7-4997-a01c-649bc7bbaeef None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:03.301919 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:03.348828 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c78cccc-53c7-4997-a01c-649bc7bbaeef None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:03.388982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747c264f-938f-4a69-a072-c5661a4df76f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:03.389635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747c264f-938f-4a69-a072-c5661a4df76f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:03.389941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747c264f-938f-4a69-a072-c5661a4df76f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:03.390171 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747c264f-938f-4a69-a072-c5661a4df76f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:03.400014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747c264f-938f-4a69-a072-c5661a4df76f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:03.403164 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747c264f-938f-4a69-a072-c5661a4df76f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:03.430114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-747c264f-938f-4a69-a072-c5661a4df76f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:19:03.430114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:03.430114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:03.430114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:03.430114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:03.430114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:03.430114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:03.430114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:03.430114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:03.430114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:03.430114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:03.430114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:03.430114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:19:03.430114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:19:03.430114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:19:03.432183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:19:03.432183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:03.432183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:03.432183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:03.432183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:03.432183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:03.432183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:03.432183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:03.432183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:03.432183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:03.432183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:03.432183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:03.432183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:03.432183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:03.432183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:03.432183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:03.432183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:03.434368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:19:03.434368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:19:03.434368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:19:03.434368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:19:03.434368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:19:03.434368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:19:03.434368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:19:03.434368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:19:03.434368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 614/1217] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:19:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:03.441487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623cb60f-c6df-4afe-ab30-96a46076cfb5 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:03.601990 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623cb60f-c6df-4afe-ab30-96a46076cfb5 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:03.634546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:03.652058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23742827-85b9-4fc3-9381-7b0a6047319f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:03.653003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23742827-85b9-4fc3-9381-7b0a6047319f None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:03.653381 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23742827-85b9-4fc3-9381-7b0a6047319f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:03.653819 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23742827-85b9-4fc3-9381-7b0a6047319f None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:03.654993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23742827-85b9-4fc3-9381-7b0a6047319f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:03.655586 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23742827-85b9-4fc3-9381-7b0a6047319f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:03.658177 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23742827-85b9-4fc3-9381-7b0a6047319f None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:03.681008 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 604/1218] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:19:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:19:03.688039 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747c264f-938f-4a69-a072-c5661a4df76f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:03.853101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747c264f-938f-4a69-a072-c5661a4df76f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:03.901436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:03.910599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a85209-1a19-42c1-9fb4-8be9996047e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:03.911694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a85209-1a19-42c1-9fb4-8be9996047e9 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:03.912035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a85209-1a19-42c1-9fb4-8be9996047e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:03.912353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a85209-1a19-42c1-9fb4-8be9996047e9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:03.913418 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a85209-1a19-42c1-9fb4-8be9996047e9 None admin] RBAC: Adding query filter params (role.id=3ebe3c0be8c34c5eae533311e9b44176, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=436deb3d60c54a799700986519b2ef68) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:03.913869 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a85209-1a19-42c1-9fb4-8be9996047e9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:03.915958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a85209-1a19-42c1-9fb4-8be9996047e9 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:03.927386 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 615/1219] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:19:03 2022] GET /identity/v3/role_assignments?user.id=436deb3d60c54a799700986519b2ef68&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=3ebe3c0be8c34c5eae533311e9b44176 => generated 259 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:06.711061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86c5e08-fb99-48cd-b382-c137ccbd7ed0 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:06.712591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86c5e08-fb99-48cd-b382-c137ccbd7ed0 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:06.713147 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86c5e08-fb99-48cd-b382-c137ccbd7ed0 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:06.716100 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 605/1220] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:19:06 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:19:06.723925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babddd16-e339-4b2b-b41f-acfbb4da23c1 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:06.724900 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babddd16-e339-4b2b-b41f-acfbb4da23c1 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:06.724900 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babddd16-e339-4b2b-b41f-acfbb4da23c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:06.749247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-babddd16-e339-4b2b-b41f-acfbb4da23c1 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:19:06.750591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-babddd16-e339-4b2b-b41f-acfbb4da23c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:19:06.850761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 616/1221] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:19:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:19:07.280224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86c5e08-fb99-48cd-b382-c137ccbd7ed0 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:07.451060 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86c5e08-fb99-48cd-b382-c137ccbd7ed0 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:07.500169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0c7dcb-4616-437b-a1f9-fa9488afaf6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:07.500858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0c7dcb-4616-437b-a1f9-fa9488afaf6f None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:07.501171 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0c7dcb-4616-437b-a1f9-fa9488afaf6f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:07.501445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0c7dcb-4616-437b-a1f9-fa9488afaf6f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:07.517104 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0c7dcb-4616-437b-a1f9-fa9488afaf6f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:19:07.525821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-6f0c7dcb-4616-437b-a1f9-fa9488afaf6f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:19:07.525821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:07.525821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:07.525821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:07.525821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:07.525821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:07.525821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:07.525821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:07.525821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:07.525821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:07.525821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:07.525821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:07.525821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:19:07.525821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:19:07.525821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:19:07.525821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:19:07.527349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:19:07.527349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:19:07.527349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:19:07.527349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:19:07.527349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:07.527349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:07.527349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:07.527349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:07.527349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:07.527349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:07.527349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:07.527349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:07.527349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:07.527349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:07.527349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:07.527349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:07.528643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:07.528643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:07.528643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:07.528643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:07.528643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:19:07.528643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:19:07.528643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:19:07.528643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:19:07.528643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:19:07.528643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:19:07.528643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:19:07.528643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:07.528643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 606/1222] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:19:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:07.538944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-babddd16-e339-4b2b-b41f-acfbb4da23c1 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:07.690044 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-babddd16-e339-4b2b-b41f-acfbb4da23c1 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:07.733242 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:07.750848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a72e36a-386d-47ae-901a-5a944293db07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:07.751745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a72e36a-386d-47ae-901a-5a944293db07 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:07.752082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a72e36a-386d-47ae-901a-5a944293db07 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:07.752533 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a72e36a-386d-47ae-901a-5a944293db07 None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:07.754076 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a72e36a-386d-47ae-901a-5a944293db07 None admin] RBAC: Adding query filter params (name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:07.754499 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a72e36a-386d-47ae-901a-5a944293db07 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:07.756468 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a72e36a-386d-47ae-901a-5a944293db07 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:07.768098 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 617/1223] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:19:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:19:07.775699 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0c7dcb-4616-437b-a1f9-fa9488afaf6f None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:07.858707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:07.955686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0c7dcb-4616-437b-a1f9-fa9488afaf6f None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:08.009855 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b306467e-79a8-4316-a032-20bcb2712319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:08.010682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b306467e-79a8-4316-a032-20bcb2712319 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:08.011199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b306467e-79a8-4316-a032-20bcb2712319 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:08.011566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b306467e-79a8-4316-a032-20bcb2712319 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:08.028981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b306467e-79a8-4316-a032-20bcb2712319 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:08.030624 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b306467e-79a8-4316-a032-20bcb2712319 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:08.052366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-b306467e-79a8-4316-a032-20bcb2712319 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:08.052366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:08.052366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:08.052366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:08.052366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:08.052366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:08.052366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:08.052366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:08.052366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:08.052366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:08.052366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:08.052366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:08.052366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:19:08.052366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:19:08.052366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:19:08.052366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:19:08.054378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:08.054378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:08.054378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:08.054378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:08.054378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:08.054378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:08.054378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:08.054378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:08.054378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:08.054378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:08.054378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:08.054378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:08.054378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:08.054378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:08.054378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:08.054378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:08.054378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:19:08.056278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:19:08.056278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:08.056278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:08.056278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 607/1224] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:08.063929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a72e36a-386d-47ae-901a-5a944293db07 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:08.259360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a72e36a-386d-47ae-901a-5a944293db07 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:08.320134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f6d152-645c-44ed-89d3-11ddbdfe8df8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:08.320916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f6d152-645c-44ed-89d3-11ddbdfe8df8 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:08.321250 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f6d152-645c-44ed-89d3-11ddbdfe8df8 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:08.321648 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f6d152-645c-44ed-89d3-11ddbdfe8df8 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:08.322417 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f6d152-645c-44ed-89d3-11ddbdfe8df8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:08.322798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f6d152-645c-44ed-89d3-11ddbdfe8df8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:08.324338 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f6d152-645c-44ed-89d3-11ddbdfe8df8 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:08.340196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 618/1225] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:19:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:08.346341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b306467e-79a8-4316-a032-20bcb2712319 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:08.488554 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:08.499771 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b306467e-79a8-4316-a032-20bcb2712319 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:08.548935 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c206d6b-7125-45b8-84c6-88fd9aa771b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:08.549627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c206d6b-7125-45b8-84c6-88fd9aa771b9 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:08.550034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c206d6b-7125-45b8-84c6-88fd9aa771b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:08.550342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c206d6b-7125-45b8-84c6-88fd9aa771b9 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:08.559671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c206d6b-7125-45b8-84c6-88fd9aa771b9 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:08.561219 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c206d6b-7125-45b8-84c6-88fd9aa771b9 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:08.580142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-3c206d6b-7125-45b8-84c6-88fd9aa771b9 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 14:19:08.580142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:08.580142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:08.580142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:08.580142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:08.580142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:08.580142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:08.580142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:08.580142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:08.580142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:08.580142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:08.580142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:08.580142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:19:08.580142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:19:08.580142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:19:08.580142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:08.583089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:08.585117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:08.585117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:08.585117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:19:08.585117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:19:08.585117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:19:08.585117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:19:08.585117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:19:08.585117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:19:08.585117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 14:19:08.585117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:08.585117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 608/1226] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:08 2022] GET /identity/v3/users/placement => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:08.589192 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f6d152-645c-44ed-89d3-11ddbdfe8df8 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:08.665915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:08.768293 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f6d152-645c-44ed-89d3-11ddbdfe8df8 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:08.833840 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8decd67f-11d3-4a9c-9cd1-cd4df21d4451 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:08.834614 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8decd67f-11d3-4a9c-9cd1-cd4df21d4451 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:08.835008 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8decd67f-11d3-4a9c-9cd1-cd4df21d4451 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:08.835342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8decd67f-11d3-4a9c-9cd1-cd4df21d4451 None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:08.836478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8decd67f-11d3-4a9c-9cd1-cd4df21d4451 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:08.836930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8decd67f-11d3-4a9c-9cd1-cd4df21d4451 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:08.839293 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8decd67f-11d3-4a9c-9cd1-cd4df21d4451 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:08.872613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 619/1227] 10.222.0.64 () {56 vars in 1143 bytes} [Tue Oct 4 14:19:08 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:19:08.880766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c206d6b-7125-45b8-84c6-88fd9aa771b9 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:09.062224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c206d6b-7125-45b8-84c6-88fd9aa771b9 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:09.134307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53129011-10e1-4013-bd5d-13ac718d9c0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:09.135136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53129011-10e1-4013-bd5d-13ac718d9c0c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:09.135529 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53129011-10e1-4013-bd5d-13ac718d9c0c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:09.135921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53129011-10e1-4013-bd5d-13ac718d9c0c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:09.149770 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53129011-10e1-4013-bd5d-13ac718d9c0c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:09.151785 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53129011-10e1-4013-bd5d-13ac718d9c0c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:09.169930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-53129011-10e1-4013-bd5d-13ac718d9c0c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:09.169930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:09.169930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:09.169930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:09.169930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:09.169930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:09.169930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:09.169930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:09.169930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:09.169930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:09.169930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:09.169930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:09.169930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:19:09.169930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:19:09.169930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:19:09.169930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:19:09.172035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:09.172035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:09.172035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:09.172035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:09.172035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:09.172035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:09.172035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:09.172035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:09.172035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:09.172035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:09.172035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:09.172035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:09.172035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:09.172035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:09.172035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:09.172035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:09.172035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:19:09.173888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:19:09.173888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:09.173888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:09.173888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 609/1228] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:19:09.180982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8decd67f-11d3-4a9c-9cd1-cd4df21d4451 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:09.344079 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8decd67f-11d3-4a9c-9cd1-cd4df21d4451 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:09.347543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:09.398905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de1447b-86a7-452a-b111-d9c8f5f1fc8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:09.399431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1447b-86a7-452a-b111-d9c8f5f1fc8e None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:09.399854 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1447b-86a7-452a-b111-d9c8f5f1fc8e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:09.400517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1447b-86a7-452a-b111-d9c8f5f1fc8e None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:09.401314 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de1447b-86a7-452a-b111-d9c8f5f1fc8e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:09.401748 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de1447b-86a7-452a-b111-d9c8f5f1fc8e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:09.403495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de1447b-86a7-452a-b111-d9c8f5f1fc8e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:09.423835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 620/1229] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:19:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:09.430565 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53129011-10e1-4013-bd5d-13ac718d9c0c None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:09.443342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:09.592596 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53129011-10e1-4013-bd5d-13ac718d9c0c None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:09.652612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd89c39-f82b-4a96-b15f-73daef6cc3f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:09.653305 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd89c39-f82b-4a96-b15f-73daef6cc3f3 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:09.653669 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd89c39-f82b-4a96-b15f-73daef6cc3f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:09.654007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd89c39-f82b-4a96-b15f-73daef6cc3f3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:09.666360 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd89c39-f82b-4a96-b15f-73daef6cc3f3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:09.669174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd89c39-f82b-4a96-b15f-73daef6cc3f3 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:09.692130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-1dd89c39-f82b-4a96-b15f-73daef6cc3f3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:19:09.692130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:09.692130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:09.692130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:09.692130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:09.692130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:09.692130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:09.692130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:09.692130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:09.692130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:09.692130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:09.692130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:09.692130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:19:09.692130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:19:09.692130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:19:09.693975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:19:09.693975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:09.693975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:09.693975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:09.693975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:09.693975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:09.693975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:09.693975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:09.693975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:09.693975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:09.693975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:09.693975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:09.693975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:09.693975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:09.693975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:09.693975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:09.693975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:09.696061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:19:09.696061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:19:09.696061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:19:09.696061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:19:09.696061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:19:09.696061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:19:09.696061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:19:09.696061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:09.696061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 610/1230] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:19:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:09.702500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de1447b-86a7-452a-b111-d9c8f5f1fc8e None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:09.854639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de1447b-86a7-452a-b111-d9c8f5f1fc8e None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:09.897365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b435ae0d-388d-4d28-b396-0be154d09e8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:09.898059 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b435ae0d-388d-4d28-b396-0be154d09e8e None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:09.898384 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b435ae0d-388d-4d28-b396-0be154d09e8e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:09.898609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b435ae0d-388d-4d28-b396-0be154d09e8e None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:09.899695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b435ae0d-388d-4d28-b396-0be154d09e8e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:09.900284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b435ae0d-388d-4d28-b396-0be154d09e8e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:09.901785 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b435ae0d-388d-4d28-b396-0be154d09e8e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:09.922445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 621/1231] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:19:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:09.923748 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd89c39-f82b-4a96-b15f-73daef6cc3f3 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:10.031235 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:10.092163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd89c39-f82b-4a96-b15f-73daef6cc3f3 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:10.153644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0d7eb1-c29f-4b71-85c5-17e3b4101add None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:10.154500 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0d7eb1-c29f-4b71-85c5-17e3b4101add None admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:10.154911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0d7eb1-c29f-4b71-85c5-17e3b4101add None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:10.155199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0d7eb1-c29f-4b71-85c5-17e3b4101add None admin] PATH_INFO: `/v3/projects/895ece8b70ef44adadc3416f81a80731/users/436deb3d60c54a799700986519b2ef68/roles/3ebe3c0be8c34c5eae533311e9b44176` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:10.178798 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0d7eb1-c29f-4b71-85c5-17e3b4101add None admin] RBAC: Authorizing `identity:create_grant(project_id=895ece8b70ef44adadc3416f81a80731, user_id=436deb3d60c54a799700986519b2ef68, role_id=3ebe3c0be8c34c5eae533311e9b44176)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:10.181630 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0d7eb1-c29f-4b71-85c5-17e3b4101add None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:10.204739 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 611/1232] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:19:09 2022] PUT /identity/v3/projects/895ece8b70ef44adadc3416f81a80731/users/436deb3d60c54a799700986519b2ef68/roles/3ebe3c0be8c34c5eae533311e9b44176 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:19:13.351296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95dc7ae-5328-4182-902a-3a3c2fed0be6 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:13.352152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95dc7ae-5328-4182-902a-3a3c2fed0be6 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:13.352152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95dc7ae-5328-4182-902a-3a3c2fed0be6 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:13.354095 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 622/1233] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:19:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:19:13.363881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dd0073-8281-41c0-b371-c07a4027172c None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:13.364788 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dd0073-8281-41c0-b371-c07a4027172c None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:13.364788 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dd0073-8281-41c0-b371-c07a4027172c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:13.391282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-a7dd0073-8281-41c0-b371-c07a4027172c None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:19:13.393534 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a7dd0073-8281-41c0-b371-c07a4027172c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:19:13.508759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 612/1234] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:19:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:19:13.852594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95dc7ae-5328-4182-902a-3a3c2fed0be6 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:13.885942 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:14.029561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95dc7ae-5328-4182-902a-3a3c2fed0be6 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:14.098117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348f5e49-e5fc-4e4b-8b3d-1a527136196f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:14.099159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348f5e49-e5fc-4e4b-8b3d-1a527136196f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:14.099630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348f5e49-e5fc-4e4b-8b3d-1a527136196f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:14.100092 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348f5e49-e5fc-4e4b-8b3d-1a527136196f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:14.118927 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348f5e49-e5fc-4e4b-8b3d-1a527136196f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:19:14.127805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-348f5e49-e5fc-4e4b-8b3d-1a527136196f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:19:14.127805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:14.127805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:14.127805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:14.127805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:14.127805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:14.127805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:14.127805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:14.127805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:14.127805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:14.127805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:14.127805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:14.127805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:19:14.127805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:19:14.127805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:19:14.127805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:19:14.131726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:19:14.131726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:19:14.131726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:19:14.131726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:19:14.131726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:14.131726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:14.131726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:14.131726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:14.131726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:14.131726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:14.131726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:14.131726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:14.131726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:14.131726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:14.131726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:14.131726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:14.133711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:14.133711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:14.133711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:14.133711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:14.133711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:19:14.133711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:19:14.133711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:19:14.133711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:19:14.133711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:19:14.133711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:19:14.133711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 14:19:14.133711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:19:14.133711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 623/1235] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Oct 4 14:19:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:14.139198 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dd0073-8281-41c0-b371-c07a4027172c None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:14.216658 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:14.315436 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dd0073-8281-41c0-b371-c07a4027172c None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:14.381626 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ded4a1b-7b9a-42ee-9930-1fc69e9eefce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:14.382629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ded4a1b-7b9a-42ee-9930-1fc69e9eefce None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:14.383112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ded4a1b-7b9a-42ee-9930-1fc69e9eefce None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:14.383591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ded4a1b-7b9a-42ee-9930-1fc69e9eefce None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:14.385071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ded4a1b-7b9a-42ee-9930-1fc69e9eefce None admin] RBAC: Adding query filter params (name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:14.385629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ded4a1b-7b9a-42ee-9930-1fc69e9eefce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:14.388467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ded4a1b-7b9a-42ee-9930-1fc69e9eefce None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:14.409439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 613/1236] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Oct 4 14:19:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:19:14.416937 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348f5e49-e5fc-4e4b-8b3d-1a527136196f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:14.536383 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:14.557764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348f5e49-e5fc-4e4b-8b3d-1a527136196f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:14.625688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e3e844-2501-4262-bbd4-7714e3cd4c8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:14.626339 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e3e844-2501-4262-bbd4-7714e3cd4c8f None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:14.626707 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e3e844-2501-4262-bbd4-7714e3cd4c8f None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:14.626873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e3e844-2501-4262-bbd4-7714e3cd4c8f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:14.639513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e3e844-2501-4262-bbd4-7714e3cd4c8f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:14.641022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e3e844-2501-4262-bbd4-7714e3cd4c8f None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:14.658328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-f1e3e844-2501-4262-bbd4-7714e3cd4c8f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:14.658328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:14.658328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:14.658328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:14.658328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:14.658328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:14.658328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:14.658328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:14.658328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:14.658328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:14.658328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:14.658328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:14.658328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:19:14.658328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:19:14.658328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:19:14.658328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:19:14.660650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:14.660650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:14.660650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:14.660650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:14.660650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:14.660650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:14.660650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:14.660650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:14.660650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:14.660650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:14.660650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:14.660650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:14.660650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:14.660650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:14.660650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:14.660650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:14.660650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:19:14.662221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:19:14.662221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:14.662221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:19:14.662221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 624/1237] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:14.669318 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ded4a1b-7b9a-42ee-9930-1fc69e9eefce None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:14.808812 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ded4a1b-7b9a-42ee-9930-1fc69e9eefce None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:14.824194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:14.860942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16aad8ee-7749-4977-bb07-344bce701e87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:14.863363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16aad8ee-7749-4977-bb07-344bce701e87 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:14.863363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16aad8ee-7749-4977-bb07-344bce701e87 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:14.863363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16aad8ee-7749-4977-bb07-344bce701e87 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:14.863363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16aad8ee-7749-4977-bb07-344bce701e87 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:14.864008 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16aad8ee-7749-4977-bb07-344bce701e87 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:14.866342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16aad8ee-7749-4977-bb07-344bce701e87 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:14.889135 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 614/1238] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:19:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:14.896645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e3e844-2501-4262-bbd4-7714e3cd4c8f None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:15.034835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e3e844-2501-4262-bbd4-7714e3cd4c8f None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:15.082540 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61227515-2642-413e-a45b-964e9bc253e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:15.083238 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61227515-2642-413e-a45b-964e9bc253e7 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:15.083505 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61227515-2642-413e-a45b-964e9bc253e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:15.083858 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61227515-2642-413e-a45b-964e9bc253e7 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:15.087966 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:15.095239 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61227515-2642-413e-a45b-964e9bc253e7 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:15.097367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61227515-2642-413e-a45b-964e9bc253e7 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:15.116337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-61227515-2642-413e-a45b-964e9bc253e7 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 14:19:15.116337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:15.116337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:15.116337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:15.116337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:15.116337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:15.116337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:15.116337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:15.116337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:15.116337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:15.116337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:15.116337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:15.116337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:19:15.116337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:19:15.116337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:19:15.116337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:15.117706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:15.119155 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:15.119155 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:15.119155 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:19:15.119155 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:19:15.119155 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:19:15.119155 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:19:15.119155 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:19:15.119155 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:19:15.119155 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 14:19:15.119155 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:19:15.119155 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 625/1239] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:14 2022] GET /identity/v3/users/placement => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:15.130560 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16aad8ee-7749-4977-bb07-344bce701e87 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:15.281244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16aad8ee-7749-4977-bb07-344bce701e87 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:15.340062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dadb6b-4f41-4893-8484-ab513b3aedee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:15.341129 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dadb6b-4f41-4893-8484-ab513b3aedee None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:15.341616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dadb6b-4f41-4893-8484-ab513b3aedee None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:15.342075 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dadb6b-4f41-4893-8484-ab513b3aedee None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:15.343279 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79dadb6b-4f41-4893-8484-ab513b3aedee None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:15.343716 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79dadb6b-4f41-4893-8484-ab513b3aedee None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:15.345739 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79dadb6b-4f41-4893-8484-ab513b3aedee None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:15.375038 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 615/1240] 10.222.0.64 () {56 vars in 1143 bytes} [Tue Oct 4 14:19:15 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:15.383822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61227515-2642-413e-a45b-964e9bc253e7 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:15.540649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61227515-2642-413e-a45b-964e9bc253e7 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:15.596174 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8242670-c4bb-4084-8285-990234d0dee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:15.597410 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8242670-c4bb-4084-8285-990234d0dee2 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:15.597746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8242670-c4bb-4084-8285-990234d0dee2 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:15.598019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8242670-c4bb-4084-8285-990234d0dee2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:15.699632 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:15.713788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8242670-c4bb-4084-8285-990234d0dee2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:15.715752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8242670-c4bb-4084-8285-990234d0dee2 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:15.877054 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 626/1241] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:19:15 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 494 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 04 14:19:15.887015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dadb6b-4f41-4893-8484-ab513b3aedee None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:15.924271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:16.051468 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dadb6b-4f41-4893-8484-ab513b3aedee None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:16.116073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de88f4a5-950b-4bc7-aea1-8db3a33e73ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:16.117335 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de88f4a5-950b-4bc7-aea1-8db3a33e73ec None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:16.117813 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de88f4a5-950b-4bc7-aea1-8db3a33e73ec None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:16.118264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de88f4a5-950b-4bc7-aea1-8db3a33e73ec None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:16.140964 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de88f4a5-950b-4bc7-aea1-8db3a33e73ec None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:16.143736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de88f4a5-950b-4bc7-aea1-8db3a33e73ec None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:16.163558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-de88f4a5-950b-4bc7-aea1-8db3a33e73ec None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:16.163558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:16.163558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:16.163558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:16.163558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:16.163558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:16.163558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:16.163558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:16.163558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:16.163558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:16.163558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:16.163558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:16.163558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:19:16.163558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:19:16.163558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:19:16.163558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:19:16.165838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:16.165838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:16.165838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:16.165838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:16.165838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:16.165838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:16.165838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:16.165838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:16.165838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:16.165838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:16.165838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:16.165838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:16.165838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:16.165838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:16.165838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:16.165838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:16.165838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:19:16.167408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:19:16.167408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:16.167408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:16.167408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 616/1242] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:16.174009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8242670-c4bb-4084-8285-990234d0dee2 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:16.315429 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:16.329135 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8242670-c4bb-4084-8285-990234d0dee2 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:16.390890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee339e9-5eac-4057-b833-9f9147067d73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:16.391893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee339e9-5eac-4057-b833-9f9147067d73 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:16.392260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee339e9-5eac-4057-b833-9f9147067d73 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:16.392631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee339e9-5eac-4057-b833-9f9147067d73 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:16.393600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee339e9-5eac-4057-b833-9f9147067d73 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:16.394514 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee339e9-5eac-4057-b833-9f9147067d73 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:16.396433 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee339e9-5eac-4057-b833-9f9147067d73 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:16.415018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 627/1243] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:19:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:16.422522 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de88f4a5-950b-4bc7-aea1-8db3a33e73ec None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:16.527283 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:16.565902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de88f4a5-950b-4bc7-aea1-8db3a33e73ec None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:16.626280 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c34ec1d-a4fa-4177-996a-c12166168855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:16.627165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c34ec1d-a4fa-4177-996a-c12166168855 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:16.627500 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c34ec1d-a4fa-4177-996a-c12166168855 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:16.627919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c34ec1d-a4fa-4177-996a-c12166168855 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:16.640610 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c34ec1d-a4fa-4177-996a-c12166168855 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:16.643021 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c34ec1d-a4fa-4177-996a-c12166168855 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:16.660912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-1c34ec1d-a4fa-4177-996a-c12166168855 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:19:16.660912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:16.660912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:16.660912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:16.660912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:16.660912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:16.660912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:16.660912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:16.660912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:16.660912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:16.660912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:16.660912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:16.660912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:19:16.660912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:19:16.660912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:19:16.662976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:19:16.662976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:16.662976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:16.662976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:16.662976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:16.662976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:16.662976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:16.662976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:16.662976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:16.662976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:16.662976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:16.662976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:16.662976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:16.662976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:16.662976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:16.662976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:16.662976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:16.664196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:19:16.664196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:19:16.664196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:19:16.664196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:19:16.664196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:19:16.664196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:19:16.664196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:19:16.664196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:16.664196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 617/1244] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:19:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:16.670088 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee339e9-5eac-4057-b833-9f9147067d73 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:16.843078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee339e9-5eac-4057-b833-9f9147067d73 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:16.890333 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8650d551-779c-4e37-b494-20ae9d6e2d9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:16.891035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8650d551-779c-4e37-b494-20ae9d6e2d9e None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:16.891311 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8650d551-779c-4e37-b494-20ae9d6e2d9e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:16.891587 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8650d551-779c-4e37-b494-20ae9d6e2d9e None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:16.892295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8650d551-779c-4e37-b494-20ae9d6e2d9e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:16.892638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8650d551-779c-4e37-b494-20ae9d6e2d9e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:16.894909 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8650d551-779c-4e37-b494-20ae9d6e2d9e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:16.909610 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 628/1245] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:19:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:19:16.917386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c34ec1d-a4fa-4177-996a-c12166168855 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:17.096568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c34ec1d-a4fa-4177-996a-c12166168855 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:17.162424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5155fe-b7e1-41f0-a1c9-6213a704d3ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:17.163355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5155fe-b7e1-41f0-a1c9-6213a704d3ed None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:17.163831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5155fe-b7e1-41f0-a1c9-6213a704d3ed None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:17.164348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5155fe-b7e1-41f0-a1c9-6213a704d3ed None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:17.165678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5155fe-b7e1-41f0-a1c9-6213a704d3ed None admin] RBAC: Adding query filter params (role.id=3ebe3c0be8c34c5eae533311e9b44176, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=436deb3d60c54a799700986519b2ef68) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:17.166215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5155fe-b7e1-41f0-a1c9-6213a704d3ed None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:17.168737 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5155fe-b7e1-41f0-a1c9-6213a704d3ed None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:17.177881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 618/1246] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:19:16 2022] GET /identity/v3/role_assignments?user.id=436deb3d60c54a799700986519b2ef68&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=3ebe3c0be8c34c5eae533311e9b44176 => generated 608 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:19:19.936661 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d75271-b41b-4c63-9d95-6ac1c5b13612 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:19.937224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d75271-b41b-4c63-9d95-6ac1c5b13612 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:19.937493 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d75271-b41b-4c63-9d95-6ac1c5b13612 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:19.940089 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 629/1247] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:19:19 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:19:19.950013 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced7ce8a-720a-4afe-9c10-dae670130d02 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:19.950865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced7ce8a-720a-4afe-9c10-dae670130d02 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:19.951142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced7ce8a-720a-4afe-9c10-dae670130d02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:19.960953 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:19.978235 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-ced7ce8a-720a-4afe-9c10-dae670130d02 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:19:19.979830 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ced7ce8a-720a-4afe-9c10-dae670130d02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:19:20.117937 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 619/1248] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:19:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:19:20.641679 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d75271-b41b-4c63-9d95-6ac1c5b13612 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:20.681194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:20.813154 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d75271-b41b-4c63-9d95-6ac1c5b13612 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:20.867158 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959f3224-6cb3-4bdb-ad43-df51039eaf0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:20.868114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959f3224-6cb3-4bdb-ad43-df51039eaf0d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:20.868671 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959f3224-6cb3-4bdb-ad43-df51039eaf0d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:20.869098 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959f3224-6cb3-4bdb-ad43-df51039eaf0d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:20.883523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959f3224-6cb3-4bdb-ad43-df51039eaf0d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:19:20.889537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-959f3224-6cb3-4bdb-ad43-df51039eaf0d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:19:20.889537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:20.889537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:20.889537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:20.889537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:20.889537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:20.889537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:20.889537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:20.889537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:20.889537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:20.889537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:20.889537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:20.889537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:19:20.889537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:19:20.889537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:19:20.889537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:19:20.890728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:19:20.890728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:19:20.890728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:19:20.890728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:19:20.890728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:20.890728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:20.890728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:20.890728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:20.890728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:20.890728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:20.890728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:20.890728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:20.890728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:20.890728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:20.890728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:20.890728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:20.891694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:20.891694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:20.891694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:20.891694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:20.891694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:19:20.891694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:19:20.891694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:19:20.891694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:19:20.891694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:19:20.891694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:19:20.891694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:19:20.891694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:19:20.891694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 630/1249] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:19:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:20.897828 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced7ce8a-720a-4afe-9c10-dae670130d02 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:21.057910 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced7ce8a-720a-4afe-9c10-dae670130d02 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:21.111579 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:21.121632 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f795f092-d5a8-4893-a5d6-e9c69a902b4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:21.122710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f795f092-d5a8-4893-a5d6-e9c69a902b4b None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:21.123232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f795f092-d5a8-4893-a5d6-e9c69a902b4b None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:21.123744 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f795f092-d5a8-4893-a5d6-e9c69a902b4b None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:21.125019 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f795f092-d5a8-4893-a5d6-e9c69a902b4b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:21.125585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f795f092-d5a8-4893-a5d6-e9c69a902b4b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:21.128891 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f795f092-d5a8-4893-a5d6-e9c69a902b4b None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:21.145074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 620/1250] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:19:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:21.152990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959f3224-6cb3-4bdb-ad43-df51039eaf0d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:21.278299 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:21.316782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959f3224-6cb3-4bdb-ad43-df51039eaf0d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:21.377375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b546b2c5-3a9f-4c75-b9e4-318b28d93062 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:21.379106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b546b2c5-3a9f-4c75-b9e4-318b28d93062 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:21.379466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b546b2c5-3a9f-4c75-b9e4-318b28d93062 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:21.379846 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b546b2c5-3a9f-4c75-b9e4-318b28d93062 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:21.396820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b546b2c5-3a9f-4c75-b9e4-318b28d93062 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:21.398841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b546b2c5-3a9f-4c75-b9e4-318b28d93062 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:21.428545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-b546b2c5-3a9f-4c75-b9e4-318b28d93062 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:21.428545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:21.428545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:21.428545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:21.428545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:21.428545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:21.428545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:21.428545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:21.428545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:21.428545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:21.428545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:21.428545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:21.428545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:19:21.428545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:19:21.428545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:19:21.428545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:19:21.433094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:21.433094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:21.433094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:21.433094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:21.433094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:21.433094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:21.433094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:21.433094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:21.433094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:21.433094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:21.433094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:21.433094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:21.433094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:21.433094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:21.433094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:21.433094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:21.433094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:19:21.435334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:19:21.435334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:21.435334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:19:21.435334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 631/1251] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:19:21.441026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f795f092-d5a8-4893-a5d6-e9c69a902b4b None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:21.604145 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f795f092-d5a8-4893-a5d6-e9c69a902b4b None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:21.658010 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b8cd7a-2dca-4c99-b1f8-cf9e28a8513d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:21.658745 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b8cd7a-2dca-4c99-b1f8-cf9e28a8513d None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:21.660156 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b8cd7a-2dca-4c99-b1f8-cf9e28a8513d None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:21.660642 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b8cd7a-2dca-4c99-b1f8-cf9e28a8513d None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:21.661280 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b8cd7a-2dca-4c99-b1f8-cf9e28a8513d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:21.661652 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b8cd7a-2dca-4c99-b1f8-cf9e28a8513d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:21.663744 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b8cd7a-2dca-4c99-b1f8-cf9e28a8513d None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:21.681291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 621/1252] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:19:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:21.688606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b546b2c5-3a9f-4c75-b9e4-318b28d93062 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:21.863353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b546b2c5-3a9f-4c75-b9e4-318b28d93062 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:21.907105 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:21.921691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3eb53f-e277-4f26-87d3-3454f540770d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:21.922608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3eb53f-e277-4f26-87d3-3454f540770d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:21.923077 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3eb53f-e277-4f26-87d3-3454f540770d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:21.923554 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3eb53f-e277-4f26-87d3-3454f540770d None admin] PATH_INFO: `/v3/users/placement` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:21.936278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3eb53f-e277-4f26-87d3-3454f540770d None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:21.938725 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3eb53f-e277-4f26-87d3-3454f540770d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:21.953315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-2d3eb53f-e277-4f26-87d3-3454f540770d None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 14:19:21.953315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:21.953315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:21.953315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:21.953315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:21.953315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:21.953315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:21.953315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:21.953315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:21.953315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:21.953315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:21.953315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:21.953315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:19:21.953315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:19:21.953315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:19:21.953315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:21.955294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:21.957803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:21.957803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:21.957803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:19:21.957803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:19:21.957803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:19:21.957803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:19:21.957803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:19:21.957803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:19:21.957803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 14:19:21.957803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:19:21.957803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 632/1253] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:21 2022] GET /identity/v3/users/placement => generated 87 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:19:21.966027 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b8cd7a-2dca-4c99-b1f8-cf9e28a8513d None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:22.041071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:22.119561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b8cd7a-2dca-4c99-b1f8-cf9e28a8513d None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:22.172223 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8986be-88d9-41ea-afea-8d307c80eb09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:22.173154 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8986be-88d9-41ea-afea-8d307c80eb09 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:22.173615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8986be-88d9-41ea-afea-8d307c80eb09 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:22.174140 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8986be-88d9-41ea-afea-8d307c80eb09 None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:22.175685 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8986be-88d9-41ea-afea-8d307c80eb09 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:22.175949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8986be-88d9-41ea-afea-8d307c80eb09 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:22.178548 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8986be-88d9-41ea-afea-8d307c80eb09 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:22.213033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 622/1254] 10.222.0.64 () {56 vars in 1143 bytes} [Tue Oct 4 14:19:21 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:19:22.219966 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3eb53f-e277-4f26-87d3-3454f540770d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:22.375428 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3eb53f-e277-4f26-87d3-3454f540770d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:22.427532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401ed74d-37a2-4624-b7e1-8a68112b6d65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:22.428058 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401ed74d-37a2-4624-b7e1-8a68112b6d65 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:22.428432 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401ed74d-37a2-4624-b7e1-8a68112b6d65 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:22.428647 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401ed74d-37a2-4624-b7e1-8a68112b6d65 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:22.526161 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:22.542500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401ed74d-37a2-4624-b7e1-8a68112b6d65 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:22.544128 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401ed74d-37a2-4624-b7e1-8a68112b6d65 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:22.686530 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 633/1255] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:19:22 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 467 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 04 14:19:22.694166 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8986be-88d9-41ea-afea-8d307c80eb09 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:22.842982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8986be-88d9-41ea-afea-8d307c80eb09 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:22.859408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:22.888440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d13b6c-ba0b-4032-aac9-108aae17294c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:22.889143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d13b6c-ba0b-4032-aac9-108aae17294c None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:22.889457 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d13b6c-ba0b-4032-aac9-108aae17294c None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:22.889732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d13b6c-ba0b-4032-aac9-108aae17294c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:22.904039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d13b6c-ba0b-4032-aac9-108aae17294c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:22.905644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d13b6c-ba0b-4032-aac9-108aae17294c None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:22.922168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-f8d13b6c-ba0b-4032-aac9-108aae17294c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:22.922168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:22.922168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:22.922168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:22.922168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:22.922168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:22.922168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:22.922168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:22.922168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:22.922168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:22.922168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:22.922168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:22.922168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:19:22.922168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:19:22.922168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:19:22.922168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:19:22.924097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:22.924097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:22.924097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:22.924097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:22.924097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:22.924097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:22.924097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:22.924097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:22.924097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:22.924097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:22.924097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:22.924097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:22.924097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:22.924097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:22.924097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:22.924097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:22.924097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:19:22.927266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:19:22.927266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:22.927266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:22.927266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 623/1256] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:19:22.930127 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401ed74d-37a2-4624-b7e1-8a68112b6d65 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:23.039404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:23.055514 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401ed74d-37a2-4624-b7e1-8a68112b6d65 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:23.110696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ef710d-a4eb-4e0d-813d-160a93b34009 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:23.111717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ef710d-a4eb-4e0d-813d-160a93b34009 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:23.112943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ef710d-a4eb-4e0d-813d-160a93b34009 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:23.113982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ef710d-a4eb-4e0d-813d-160a93b34009 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:23.115079 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ef710d-a4eb-4e0d-813d-160a93b34009 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:23.115478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ef710d-a4eb-4e0d-813d-160a93b34009 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:23.117325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ef710d-a4eb-4e0d-813d-160a93b34009 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:23.134941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 634/1257] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:19:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:19:23.141047 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d13b6c-ba0b-4032-aac9-108aae17294c None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:23.281133 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d13b6c-ba0b-4032-aac9-108aae17294c None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:23.329503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189fca1e-6d6c-4e41-99dd-ca062e5da941 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:23.330191 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189fca1e-6d6c-4e41-99dd-ca062e5da941 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:23.330512 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189fca1e-6d6c-4e41-99dd-ca062e5da941 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:23.331451 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189fca1e-6d6c-4e41-99dd-ca062e5da941 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:23.339304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189fca1e-6d6c-4e41-99dd-ca062e5da941 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:23.341151 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189fca1e-6d6c-4e41-99dd-ca062e5da941 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:23.354386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-189fca1e-6d6c-4e41-99dd-ca062e5da941 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:19:23.354386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:23.354386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:23.354386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:23.354386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:23.354386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:23.354386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:23.354386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:23.354386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:23.354386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:23.354386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:23.354386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:23.354386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:19:23.354386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:19:23.354386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:19:23.356174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:19:23.356174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:23.356174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:23.356174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:23.356174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:23.356174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:23.356174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:23.356174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:23.356174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:23.356174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:23.356174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:23.356174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:23.356174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:23.356174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:23.356174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:23.356174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:23.356174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:23.357585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:19:23.357585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:19:23.357585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:19:23.357585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:19:23.357585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:19:23.357585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:19:23.357585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:19:23.357585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:23.357585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 624/1258] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:19:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:19:23.364752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ef710d-a4eb-4e0d-813d-160a93b34009 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:23.487521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ef710d-a4eb-4e0d-813d-160a93b34009 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:23.525008 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914888df-009c-4ef6-ac13-10e86e33edfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:23.525618 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914888df-009c-4ef6-ac13-10e86e33edfa None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:23.525896 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914888df-009c-4ef6-ac13-10e86e33edfa None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:23.526168 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914888df-009c-4ef6-ac13-10e86e33edfa None admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:23.526866 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914888df-009c-4ef6-ac13-10e86e33edfa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:23.527165 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914888df-009c-4ef6-ac13-10e86e33edfa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:23.528640 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914888df-009c-4ef6-ac13-10e86e33edfa None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:23.543576 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 635/1259] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:19:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:23.554491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189fca1e-6d6c-4e41-99dd-ca062e5da941 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:23.584102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:23.671185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189fca1e-6d6c-4e41-99dd-ca062e5da941 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:23.711597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff45dd20-a9f2-46bf-bf55-a04e93c94008 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:23.712252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff45dd20-a9f2-46bf-bf55-a04e93c94008 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:23.712567 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff45dd20-a9f2-46bf-bf55-a04e93c94008 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:23.712846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff45dd20-a9f2-46bf-bf55-a04e93c94008 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:23.713605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff45dd20-a9f2-46bf-bf55-a04e93c94008 None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=436deb3d60c54a799700986519b2ef68) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:23.713916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff45dd20-a9f2-46bf-bf55-a04e93c94008 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:23.715498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff45dd20-a9f2-46bf-bf55-a04e93c94008 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:23.722962 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 625/1260] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:19:23 2022] GET /identity/v3/role_assignments?user.id=436deb3d60c54a799700986519b2ef68&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 259 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:25.994908 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72593c97-0f44-45c2-86a7-8e67a03305a5 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:25.995546 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72593c97-0f44-45c2-86a7-8e67a03305a5 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:25.995546 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72593c97-0f44-45c2-86a7-8e67a03305a5 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:25.996986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 636/1261] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:19:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:19:26.002720 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7185c650-2724-4d7f-bafa-39a083b868c6 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:26.003278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7185c650-2724-4d7f-bafa-39a083b868c6 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:26.003467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7185c650-2724-4d7f-bafa-39a083b868c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:26.023013 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-7185c650-2724-4d7f-bafa-39a083b868c6 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:19:26.024820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7185c650-2724-4d7f-bafa-39a083b868c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:19:26.121029 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 626/1262] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:19:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:19:26.418955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72593c97-0f44-45c2-86a7-8e67a03305a5 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:26.431589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:26.546942 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72593c97-0f44-45c2-86a7-8e67a03305a5 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:26.589847 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b051d32-c7bc-437b-8ef3-454a05f73050 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:26.590491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b051d32-c7bc-437b-8ef3-454a05f73050 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:26.590770 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b051d32-c7bc-437b-8ef3-454a05f73050 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:26.591040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b051d32-c7bc-437b-8ef3-454a05f73050 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:26.604224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b051d32-c7bc-437b-8ef3-454a05f73050 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:19:26.609266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-3b051d32-c7bc-437b-8ef3-454a05f73050 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:19:26.609266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:26.609266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:26.609266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:26.609266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:26.609266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:26.609266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:26.609266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:26.609266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:26.609266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:26.609266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:26.609266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:26.609266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:19:26.609266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:19:26.609266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:19:26.609266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:19:26.611430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:19:26.611430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:19:26.611430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:19:26.611430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:19:26.611430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:26.611430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:26.611430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:26.611430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:26.611430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:26.611430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:26.611430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:26.611430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:26.611430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:26.611430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:26.611430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:26.611430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:26.613521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:26.613521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:26.613521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:26.613521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:26.613521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:19:26.613521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:19:26.613521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:19:26.613521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:19:26.613521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:19:26.613521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:19:26.613521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:19:26.613521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:19:26.613521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 637/1263] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:19:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:26.617530 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7185c650-2724-4d7f-bafa-39a083b868c6 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:26.638537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:26.741675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7185c650-2724-4d7f-bafa-39a083b868c6 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:26.784377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd176d5f-0b6a-47f7-9fbc-177e9909c589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:26.785258 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd176d5f-0b6a-47f7-9fbc-177e9909c589 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:26.785535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd176d5f-0b6a-47f7-9fbc-177e9909c589 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:26.785809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd176d5f-0b6a-47f7-9fbc-177e9909c589 None admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:26.786501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd176d5f-0b6a-47f7-9fbc-177e9909c589 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:26.786805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd176d5f-0b6a-47f7-9fbc-177e9909c589 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:26.788219 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd176d5f-0b6a-47f7-9fbc-177e9909c589 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:26.798285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 627/1264] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:19:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:19:26.804849 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b051d32-c7bc-437b-8ef3-454a05f73050 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:26.886225 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:26.943761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b051d32-c7bc-437b-8ef3-454a05f73050 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:26.989764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5127435-ee42-4e4f-9fc6-5898c262caf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:26.990538 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5127435-ee42-4e4f-9fc6-5898c262caf7 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:26.990814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5127435-ee42-4e4f-9fc6-5898c262caf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:26.991083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5127435-ee42-4e4f-9fc6-5898c262caf7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:27.005559 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5127435-ee42-4e4f-9fc6-5898c262caf7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:27.007037 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5127435-ee42-4e4f-9fc6-5898c262caf7 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:27.026185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-b5127435-ee42-4e4f-9fc6-5898c262caf7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:27.026185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:27.026185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:27.026185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:27.026185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:27.026185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:27.026185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:27.026185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:27.026185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:27.026185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:27.026185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:27.026185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:27.026185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:19:27.026185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:19:27.026185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:19:27.026185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:19:27.028153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:27.028153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:27.028153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:27.028153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:27.028153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:27.028153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:27.028153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:27.028153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:27.028153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:27.028153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:27.028153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:27.028153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:27.028153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:27.028153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:27.028153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:27.028153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:27.028153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:19:27.030012 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:19:27.030012 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:27.030012 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:19:27.030012 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 638/1265] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:27.034988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd176d5f-0b6a-47f7-9fbc-177e9909c589 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:27.160407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:27.183573 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd176d5f-0b6a-47f7-9fbc-177e9909c589 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:27.225269 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f4ef75-b62d-484e-b043-16bfe3a5a4c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:27.225979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f4ef75-b62d-484e-b043-16bfe3a5a4c0 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:27.226305 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f4ef75-b62d-484e-b043-16bfe3a5a4c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:27.226615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f4ef75-b62d-484e-b043-16bfe3a5a4c0 None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:27.227310 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f4ef75-b62d-484e-b043-16bfe3a5a4c0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:27.227657 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f4ef75-b62d-484e-b043-16bfe3a5a4c0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:27.229153 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f4ef75-b62d-484e-b043-16bfe3a5a4c0 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:27.243785 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 628/1266] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:19:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:27.254058 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5127435-ee42-4e4f-9fc6-5898c262caf7 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:27.403514 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5127435-ee42-4e4f-9fc6-5898c262caf7 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:27.444337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:27.471041 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f368a2-609a-4b0e-8957-29835043d130 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:27.472644 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f368a2-609a-4b0e-8957-29835043d130 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:27.473123 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f368a2-609a-4b0e-8957-29835043d130 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:27.473609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f368a2-609a-4b0e-8957-29835043d130 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:27.490831 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f368a2-609a-4b0e-8957-29835043d130 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:27.493740 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f368a2-609a-4b0e-8957-29835043d130 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:27.510495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-92f368a2-609a-4b0e-8957-29835043d130 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 14:19:27.510495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:27.510495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:27.510495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:27.510495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:27.510495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:27.510495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:27.510495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:27.510495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:27.510495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:27.510495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:27.510495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:27.510495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:19:27.510495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:19:27.510495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:19:27.510495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:27.512509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:27.514479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:27.514479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:27.514479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:19:27.514479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:19:27.514479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:19:27.514479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:19:27.514479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:19:27.514479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:19:27.514479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 14:19:27.514479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:19:27.514479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 639/1267] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:27 2022] GET /identity/v3/users/placement => generated 87 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:27.519842 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f4ef75-b62d-484e-b043-16bfe3a5a4c0 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:27.634814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f4ef75-b62d-484e-b043-16bfe3a5a4c0 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:27.678946 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c377ee2-057f-4c7b-a692-860d071d6878 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:27.679475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c377ee2-057f-4c7b-a692-860d071d6878 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:27.679760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c377ee2-057f-4c7b-a692-860d071d6878 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:27.680033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c377ee2-057f-4c7b-a692-860d071d6878 None admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:27.680847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c377ee2-057f-4c7b-a692-860d071d6878 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:27.681156 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c377ee2-057f-4c7b-a692-860d071d6878 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:27.682669 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c377ee2-057f-4c7b-a692-860d071d6878 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:27.694480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:27.705539 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 629/1268] 10.222.0.64 () {56 vars in 1143 bytes} [Tue Oct 4 14:19:27 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:27.711939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f368a2-609a-4b0e-8957-29835043d130 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:27.832196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f368a2-609a-4b0e-8957-29835043d130 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:27.872229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1101148c-9399-45a1-975f-b14e4fcfd0f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:27.872904 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1101148c-9399-45a1-975f-b14e4fcfd0f9 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:27.873179 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1101148c-9399-45a1-975f-b14e4fcfd0f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:27.873446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1101148c-9399-45a1-975f-b14e4fcfd0f9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:27.886627 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1101148c-9399-45a1-975f-b14e4fcfd0f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:27.888375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1101148c-9399-45a1-975f-b14e4fcfd0f9 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:27.905853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-1101148c-9399-45a1-975f-b14e4fcfd0f9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:27.905853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:27.905853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:27.905853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:27.905853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:27.905853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:27.905853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:27.905853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:27.905853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:27.905853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:27.905853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:27.905853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:27.905853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:19:27.905853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:19:27.905853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:19:27.905853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:19:27.907328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:27.907328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:27.907328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:27.907328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:27.907328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:27.907328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:27.907328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:27.907328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:27.907328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:27.907328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:27.907328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:27.907328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:27.907328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:27.907328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:27.907328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:27.907328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:27.907328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:19:27.908337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:19:27.908337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:27.908337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:19:27.908337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 640/1269] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:19:27.915087 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c377ee2-057f-4c7b-a692-860d071d6878 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:28.102465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c377ee2-057f-4c7b-a692-860d071d6878 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:28.147648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dccb849-fed0-40e9-8878-c4293bcac07a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:28.148414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dccb849-fed0-40e9-8878-c4293bcac07a None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:28.148826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dccb849-fed0-40e9-8878-c4293bcac07a None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:28.149150 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dccb849-fed0-40e9-8878-c4293bcac07a None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:28.150225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dccb849-fed0-40e9-8878-c4293bcac07a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:28.150622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dccb849-fed0-40e9-8878-c4293bcac07a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:28.152278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dccb849-fed0-40e9-8878-c4293bcac07a None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:28.168005 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 630/1270] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:19:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:28.174176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1101148c-9399-45a1-975f-b14e4fcfd0f9 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:28.210643 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:28.304519 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1101148c-9399-45a1-975f-b14e4fcfd0f9 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:28.343561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197e67b5-3a39-40f9-a42d-7ce80c1e0d1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:28.344202 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197e67b5-3a39-40f9-a42d-7ce80c1e0d1d None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:28.344512 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197e67b5-3a39-40f9-a42d-7ce80c1e0d1d None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:28.344784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197e67b5-3a39-40f9-a42d-7ce80c1e0d1d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:28.356408 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197e67b5-3a39-40f9-a42d-7ce80c1e0d1d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:28.359141 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197e67b5-3a39-40f9-a42d-7ce80c1e0d1d None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:28.382215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-197e67b5-3a39-40f9-a42d-7ce80c1e0d1d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:19:28.382215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:28.382215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:28.382215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:28.382215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:28.382215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:28.382215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:28.382215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:28.382215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:28.382215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:28.382215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:28.382215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:28.382215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:19:28.382215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:19:28.382215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:19:28.384211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:19:28.384211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:28.384211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:28.384211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:28.384211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:28.384211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:28.384211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:28.384211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:28.384211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:28.384211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:28.384211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:28.384211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:28.384211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:28.384211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:28.384211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:28.384211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:28.384211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:28.386035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:19:28.386035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:19:28.386035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:19:28.386035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:19:28.386035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:19:28.386035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:19:28.386035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:19:28.386035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:19:28.386035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 641/1271] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:19:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:28.392704 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dccb849-fed0-40e9-8878-c4293bcac07a None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:28.475347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:28.552106 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dccb849-fed0-40e9-8878-c4293bcac07a None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:28.603745 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fb741c-5a47-4c6a-960b-e63102c11628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:28.604401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fb741c-5a47-4c6a-960b-e63102c11628 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:28.604684 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fb741c-5a47-4c6a-960b-e63102c11628 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:28.604960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fb741c-5a47-4c6a-960b-e63102c11628 None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:28.605676 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fb741c-5a47-4c6a-960b-e63102c11628 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:28.605990 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fb741c-5a47-4c6a-960b-e63102c11628 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:28.607543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fb741c-5a47-4c6a-960b-e63102c11628 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:28.622889 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 631/1272] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:19:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:19:28.631169 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197e67b5-3a39-40f9-a42d-7ce80c1e0d1d None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:28.751481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:28.770317 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197e67b5-3a39-40f9-a42d-7ce80c1e0d1d None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:28.811503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dc1ec2-a23c-4089-a9fa-3650441ba2c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:28.812168 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dc1ec2-a23c-4089-a9fa-3650441ba2c5 None admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:28.812481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dc1ec2-a23c-4089-a9fa-3650441ba2c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:28.812754 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dc1ec2-a23c-4089-a9fa-3650441ba2c5 None admin] PATH_INFO: `/v3/projects/895ece8b70ef44adadc3416f81a80731/users/436deb3d60c54a799700986519b2ef68/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:28.815431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91dc1ec2-a23c-4089-a9fa-3650441ba2c5 None admin] RBAC: Authorizing `identity:create_grant(project_id=895ece8b70ef44adadc3416f81a80731, user_id=436deb3d60c54a799700986519b2ef68, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:28.817093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91dc1ec2-a23c-4089-a9fa-3650441ba2c5 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:28.832981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 642/1273] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Oct 4 14:19:28 2022] PUT /identity/v3/projects/895ece8b70ef44adadc3416f81a80731/users/436deb3d60c54a799700986519b2ef68/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:19:31.209307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a1b528-7143-43ed-8468-438b28c8ee0d None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:31.210182 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a1b528-7143-43ed-8468-438b28c8ee0d None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:31.210182 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a1b528-7143-43ed-8468-438b28c8ee0d None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:31.211903 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 632/1274] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:19:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:19:31.217955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea0ea54-a7aa-4d1e-afb6-645000d93054 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:31.218559 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea0ea54-a7aa-4d1e-afb6-645000d93054 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:31.218559 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea0ea54-a7aa-4d1e-afb6-645000d93054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:31.239636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-5ea0ea54-a7aa-4d1e-afb6-645000d93054 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:19:31.240920 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5ea0ea54-a7aa-4d1e-afb6-645000d93054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:19:31.319858 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 643/1275] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:19:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:19:31.619972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a1b528-7143-43ed-8468-438b28c8ee0d None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:31.753711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a1b528-7143-43ed-8468-438b28c8ee0d None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:31.776934 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:31.803156 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a876dc8a-8632-40f5-a95a-de59f56a931f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:31.803839 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a876dc8a-8632-40f5-a95a-de59f56a931f None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:31.804092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a876dc8a-8632-40f5-a95a-de59f56a931f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:31.804408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a876dc8a-8632-40f5-a95a-de59f56a931f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:31.817697 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a876dc8a-8632-40f5-a95a-de59f56a931f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 14:19:31.822734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-a876dc8a-8632-40f5-a95a-de59f56a931f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:19:31.822734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:31.822734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:31.822734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:31.822734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:31.822734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:31.822734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:31.822734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:31.822734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:31.822734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:31.822734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:31.822734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:31.822734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 14:19:31.822734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 14:19:31.822734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 14:19:31.822734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 14:19:31.824462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 14:19:31.824462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 14:19:31.824462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 14:19:31.824462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 14:19:31.824462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:31.824462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:31.824462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:31.824462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:31.824462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:31.824462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:31.824462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:31.824462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:31.824462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:31.824462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:31.824462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:31.824462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:31.825759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:31.825759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:31.825759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:31.825759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:31.825759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 14:19:31.825759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 14:19:31.825759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 14:19:31.825759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 14:19:31.825759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 14:19:31.825759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 14:19:31.825759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 14:19:31.825759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:31.825759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 633/1276] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Oct 4 14:19:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:31.830318 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea0ea54-a7aa-4d1e-afb6-645000d93054 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:31.929651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:31.953894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea0ea54-a7aa-4d1e-afb6-645000d93054 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:31.996227 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d494ee-3968-4263-b8c6-77fd87bab2bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:31.996876 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d494ee-3968-4263-b8c6-77fd87bab2bd None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:31.997150 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d494ee-3968-4263-b8c6-77fd87bab2bd None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:31.997417 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d494ee-3968-4263-b8c6-77fd87bab2bd None admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:31.998106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d494ee-3968-4263-b8c6-77fd87bab2bd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:31.998407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d494ee-3968-4263-b8c6-77fd87bab2bd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:31.999798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d494ee-3968-4263-b8c6-77fd87bab2bd None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:32.010184 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 644/1277] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Oct 4 14:19:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:19:32.020083 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a876dc8a-8632-40f5-a95a-de59f56a931f None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:32.139110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a876dc8a-8632-40f5-a95a-de59f56a931f None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:32.180418 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45593458-47ef-4f08-83e6-160f441f7645 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:32.180991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45593458-47ef-4f08-83e6-160f441f7645 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:32.181275 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45593458-47ef-4f08-83e6-160f441f7645 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:32.181549 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45593458-47ef-4f08-83e6-160f441f7645 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:32.194369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45593458-47ef-4f08-83e6-160f441f7645 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:32.195952 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45593458-47ef-4f08-83e6-160f441f7645 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:32.212067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-45593458-47ef-4f08-83e6-160f441f7645 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:32.212067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:32.212067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:32.212067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:32.212067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:32.212067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:32.212067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:32.212067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:32.212067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:32.212067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:32.212067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:32.212067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:32.212067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:19:32.212067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:19:32.212067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:19:32.212067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:19:32.214402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:32.214402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:32.214402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:32.214402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:32.214402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:32.214402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:32.214402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:32.214402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:32.214402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:32.214402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:32.214402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:32.214402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:32.214402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:32.214402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:32.214402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:32.214402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:32.214402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:19:32.216110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:19:32.216110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:32.216110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:32.216110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 634/1278] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:32.222724 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d494ee-3968-4263-b8c6-77fd87bab2bd None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:32.352582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d494ee-3968-4263-b8c6-77fd87bab2bd None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:32.391028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42f444c-eb91-43a2-b81f-010975531327 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:32.391708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42f444c-eb91-43a2-b81f-010975531327 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:32.392026 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42f444c-eb91-43a2-b81f-010975531327 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:32.392365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42f444c-eb91-43a2-b81f-010975531327 None admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:32.393641 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42f444c-eb91-43a2-b81f-010975531327 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:32.393984 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42f444c-eb91-43a2-b81f-010975531327 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:32.395653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42f444c-eb91-43a2-b81f-010975531327 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:32.410485 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 645/1279] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:19:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:19:32.416762 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45593458-47ef-4f08-83e6-160f441f7645 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:32.452582 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:32.538718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45593458-47ef-4f08-83e6-160f441f7645 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:32.612513 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a617c2cd-1657-4fe2-84b0-ac6c2719b38e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:32.613232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a617c2cd-1657-4fe2-84b0-ac6c2719b38e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:32.613565 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a617c2cd-1657-4fe2-84b0-ac6c2719b38e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:32.613886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a617c2cd-1657-4fe2-84b0-ac6c2719b38e None admin] PATH_INFO: `/v3/users/placement` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:32.622877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a617c2cd-1657-4fe2-84b0-ac6c2719b38e None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:32.624462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a617c2cd-1657-4fe2-84b0-ac6c2719b38e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:32.637831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-a617c2cd-1657-4fe2-84b0-ac6c2719b38e None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 14:19:32.637831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:32.637831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:32.637831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:32.637831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:32.637831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:32.637831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:32.637831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:32.637831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:32.637831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:32.637831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:32.637831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:32.637831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 14:19:32.637831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 14:19:32.637831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 14:19:32.637831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:32.639650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:32.641553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:32.641553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:32.641553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 14:19:32.641553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 14:19:32.641553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 14:19:32.641553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 14:19:32.641553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 14:19:32.641553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 14:19:32.641553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 14:19:32.641553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:32.641553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 635/1280] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:32 2022] GET /identity/v3/users/placement => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:32.647281 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42f444c-eb91-43a2-b81f-010975531327 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:32.651771 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:32.765612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42f444c-eb91-43a2-b81f-010975531327 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:32.807133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2caa0093-d637-456e-815f-09fb0ece972e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:32.807856 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caa0093-d637-456e-815f-09fb0ece972e None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:32.808200 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caa0093-d637-456e-815f-09fb0ece972e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:32.808645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caa0093-d637-456e-815f-09fb0ece972e None admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:32.809550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2caa0093-d637-456e-815f-09fb0ece972e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:32.809947 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2caa0093-d637-456e-815f-09fb0ece972e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:32.811491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2caa0093-d637-456e-815f-09fb0ece972e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:32.830982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 646/1281] 10.222.0.64 () {56 vars in 1143 bytes} [Tue Oct 4 14:19:32 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:32.837925 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a617c2cd-1657-4fe2-84b0-ac6c2719b38e None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:33.019805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:33.037134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a617c2cd-1657-4fe2-84b0-ac6c2719b38e None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:33.099476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53f2c96-2591-4c74-b2c3-cd92e33bb68b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:33.100565 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53f2c96-2591-4c74-b2c3-cd92e33bb68b None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:33.101077 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53f2c96-2591-4c74-b2c3-cd92e33bb68b None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:33.101568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53f2c96-2591-4c74-b2c3-cd92e33bb68b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:33.184769 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53f2c96-2591-4c74-b2c3-cd92e33bb68b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:33.186239 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53f2c96-2591-4c74-b2c3-cd92e33bb68b None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:33.296126 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:33.308840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 636/1282] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Oct 4 14:19:32 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 472 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 14:19:33.316548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2caa0093-d637-456e-815f-09fb0ece972e None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:33.390299 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:33.462051 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2caa0093-d637-456e-815f-09fb0ece972e None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:33.504760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8546e3-8c70-4dad-ac2f-4d5724af12fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:33.505447 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8546e3-8c70-4dad-ac2f-4d5724af12fc None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:33.505766 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8546e3-8c70-4dad-ac2f-4d5724af12fc None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:33.506077 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8546e3-8c70-4dad-ac2f-4d5724af12fc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:33.518150 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8546e3-8c70-4dad-ac2f-4d5724af12fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:33.519810 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8546e3-8c70-4dad-ac2f-4d5724af12fc None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:33.537751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-5d8546e3-8c70-4dad-ac2f-4d5724af12fc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:33.537751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:33.537751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:33.537751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:33.537751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:33.537751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:33.537751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:33.537751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:33.537751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:33.537751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:33.537751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:33.537751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:33.537751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 14:19:33.537751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 14:19:33.537751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 14:19:33.537751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 14:19:33.538862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:33.538862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:33.538862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:33.538862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:33.538862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:33.538862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:33.538862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:33.538862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:33.538862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:33.538862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:33.538862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:33.538862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:33.538862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:33.538862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:33.538862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:33.538862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:33.538862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 14:19:33.540507 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 14:19:33.540507 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 14:19:33.540507 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:19:33.540507 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 647/1283] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:19:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 14:19:33.545841 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53f2c96-2591-4c74-b2c3-cd92e33bb68b None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:33.663727 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53f2c96-2591-4c74-b2c3-cd92e33bb68b None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:33.705749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd44bc3a-dc2d-4ebd-8602-e8eda2f46d2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:33.706453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd44bc3a-dc2d-4ebd-8602-e8eda2f46d2f None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:33.706732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd44bc3a-dc2d-4ebd-8602-e8eda2f46d2f None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:33.707006 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd44bc3a-dc2d-4ebd-8602-e8eda2f46d2f None admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:33.707667 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd44bc3a-dc2d-4ebd-8602-e8eda2f46d2f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:33.707970 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd44bc3a-dc2d-4ebd-8602-e8eda2f46d2f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:33.709434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd44bc3a-dc2d-4ebd-8602-e8eda2f46d2f None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:33.723159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 637/1284] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Oct 4 14:19:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:33.729644 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8546e3-8c70-4dad-ac2f-4d5724af12fc None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:33.851217 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8546e3-8c70-4dad-ac2f-4d5724af12fc None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:33.867833 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:33.899055 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c74a82f-f58e-4bf1-ae0f-1bb283b39c67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:33.900646 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c74a82f-f58e-4bf1-ae0f-1bb283b39c67 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:33.900924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c74a82f-f58e-4bf1-ae0f-1bb283b39c67 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:33.901192 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c74a82f-f58e-4bf1-ae0f-1bb283b39c67 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:33.912040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c74a82f-f58e-4bf1-ae0f-1bb283b39c67 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:33.913578 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c74a82f-f58e-4bf1-ae0f-1bb283b39c67 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:33.928481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application [None req-3c74a82f-f58e-4bf1-ae0f-1bb283b39c67 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:19:33.928481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:33.928481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:33.928481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:33.928481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:33.928481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:33.928481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:33.928481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:33.928481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:33.928481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:33.928481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:33.928481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:33.928481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 14:19:33.928481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 14:19:33.928481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 14:19:33.930595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 14:19:33.930595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:33.930595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:33.930595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:33.930595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:33.930595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:33.930595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:33.930595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:33.930595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:33.930595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:33.930595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:33.930595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:33.930595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:33.930595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:33.930595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:33.930595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:33.930595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:33.932738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 14:19:33.932738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 14:19:33.932738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 14:19:33.932738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 14:19:33.932738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 14:19:33.932738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 14:19:33.932738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 14:19:33.932738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: ERROR keystone.server.flask.application Oct 04 14:19:33.932738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 648/1285] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Oct 4 14:19:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:33.936820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd44bc3a-dc2d-4ebd-8602-e8eda2f46d2f None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:33.995595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:34.055575 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd44bc3a-dc2d-4ebd-8602-e8eda2f46d2f None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:34.098258 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bc0eb0-9449-4af5-b9c8-4b906a083925 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:34.098879 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bc0eb0-9449-4af5-b9c8-4b906a083925 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:34.099156 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bc0eb0-9449-4af5-b9c8-4b906a083925 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:34.099427 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bc0eb0-9449-4af5-b9c8-4b906a083925 None admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:34.100130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43bc0eb0-9449-4af5-b9c8-4b906a083925 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:34.100466 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43bc0eb0-9449-4af5-b9c8-4b906a083925 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:34.101923 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43bc0eb0-9449-4af5-b9c8-4b906a083925 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:34.116443 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 638/1286] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Oct 4 14:19:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:34.125942 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c74a82f-f58e-4bf1-ae0f-1bb283b39c67 None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:34.251884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c74a82f-f58e-4bf1-ae0f-1bb283b39c67 None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:34.292015 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b27a0a-757f-45c6-acdb-48bfcde836b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:34.292717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b27a0a-757f-45c6-acdb-48bfcde836b9 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:34.293038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b27a0a-757f-45c6-acdb-48bfcde836b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:34.293352 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b27a0a-757f-45c6-acdb-48bfcde836b9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:34.294142 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b27a0a-757f-45c6-acdb-48bfcde836b9 None admin] RBAC: Adding query filter params (role.id=7c36efa05ff1457d9f33747e09f2c260, scope.project.id=895ece8b70ef44adadc3416f81a80731, user.id=436deb3d60c54a799700986519b2ef68) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:34.294487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b27a0a-757f-45c6-acdb-48bfcde836b9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:34.295983 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b27a0a-757f-45c6-acdb-48bfcde836b9 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:34.303363 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 649/1287] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Oct 4 14:19:34 2022] GET /identity/v3/role_assignments?user.id=436deb3d60c54a799700986519b2ef68&scope.project.id=895ece8b70ef44adadc3416f81a80731&role.id=7c36efa05ff1457d9f33747e09f2c260 => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:36.673036 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89254dc7-cb8b-418d-8a57-36bc8367079e None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:36.673635 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89254dc7-cb8b-418d-8a57-36bc8367079e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:36.673635 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89254dc7-cb8b-418d-8a57-36bc8367079e None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:36.675564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 639/1288] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:19:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:19:36.681703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50716bc0-0e0f-416b-8b5b-8f85900379f5 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:36.682266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50716bc0-0e0f-416b-8b5b-8f85900379f5 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:36.682266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50716bc0-0e0f-416b-8b5b-8f85900379f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:36.702973 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-50716bc0-0e0f-416b-8b5b-8f85900379f5 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:19:36.704101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-50716bc0-0e0f-416b-8b5b-8f85900379f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:19:36.741573 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:36.789740 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 650/1289] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:19:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:19:37.084352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89254dc7-cb8b-418d-8a57-36bc8367079e None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:37.218152 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89254dc7-cb8b-418d-8a57-36bc8367079e None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:37.231702 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:37.261006 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5168b8-c1bd-468b-8640-44e4fdfc7199 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:37.261717 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5168b8-c1bd-468b-8640-44e4fdfc7199 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:37.262044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5168b8-c1bd-468b-8640-44e4fdfc7199 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:37.262362 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5168b8-c1bd-468b-8640-44e4fdfc7199 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:37.263011 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5168b8-c1bd-468b-8640-44e4fdfc7199 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:37.264480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5168b8-c1bd-468b-8640-44e4fdfc7199 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:37.275591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-af5168b8-c1bd-468b-8640-44e4fdfc7199 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 14:19:37.275591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:37.275591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:19:37.275591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:19:37.275591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:19:37.275591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:19:37.275591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:19:37.275591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:19:37.275591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 14:19:37.275591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:37.275591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:19:37.275591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:37.275591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:37.275591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:37.275591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:37.275591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:37.275591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:37.277561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:37.277561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:37.277561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:37.277561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:37.277561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:37.277561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:37.277561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:19:37.277561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:19:37.277561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:19:37.277561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:19:37.277561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:37.277561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:37.277561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:37.277561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:37.277561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:37.277561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:37.279347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:37.279347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:37.279347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:37.279347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:37.279347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:37.279347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:37.279347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:37.279347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:37.279347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:37.279347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:37.279347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:19:37.279347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:19:37.279347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 14:19:37.279347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:37.279347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 640/1290] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Oct 4 14:19:37 2022] GET /identity/v3/services/placement => generated 90 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:37.285161 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50716bc0-0e0f-416b-8b5b-8f85900379f5 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:37.401846 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50716bc0-0e0f-416b-8b5b-8f85900379f5 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:37.439962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7064b89-bc25-4a75-b527-9f2b31f3d3bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:37.440685 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7064b89-bc25-4a75-b527-9f2b31f3d3bc None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:37.441074 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7064b89-bc25-4a75-b527-9f2b31f3d3bc None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:37.441352 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7064b89-bc25-4a75-b527-9f2b31f3d3bc None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:37.441988 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7064b89-bc25-4a75-b527-9f2b31f3d3bc None admin] RBAC: Adding query filter params (name=placement) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:37.442289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7064b89-bc25-4a75-b527-9f2b31f3d3bc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:37.443652 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7064b89-bc25-4a75-b527-9f2b31f3d3bc None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:37.449212 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 651/1291] 10.222.0.64 () {56 vars in 1116 bytes} [Tue Oct 4 14:19:37 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:37.457017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5168b8-c1bd-468b-8640-44e4fdfc7199 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:37.576965 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5168b8-c1bd-468b-8640-44e4fdfc7199 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:37.616099 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d563d8-caef-4ba5-b3fc-1fd44f4fed0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:37.616822 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d563d8-caef-4ba5-b3fc-1fd44f4fed0e None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:37.617151 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d563d8-caef-4ba5-b3fc-1fd44f4fed0e None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:37.617497 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d563d8-caef-4ba5-b3fc-1fd44f4fed0e None admin] PATH_INFO: `/v3/services` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:37.618185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d563d8-caef-4ba5-b3fc-1fd44f4fed0e None admin] RBAC: Adding query filter params (type=placement) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:37.618545 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d563d8-caef-4ba5-b3fc-1fd44f4fed0e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:37.619995 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d563d8-caef-4ba5-b3fc-1fd44f4fed0e None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:37.625832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 641/1292] 10.222.0.64 () {56 vars in 1116 bytes} [Tue Oct 4 14:19:37 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:39.993252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1df969-ef71-4508-aed6-92dd697221ba None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:39.993896 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1df969-ef71-4508-aed6-92dd697221ba None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:39.993896 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1df969-ef71-4508-aed6-92dd697221ba None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:40.002461 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 652/1293] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:19:39 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:19:40.009570 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52324442-5fdb-49fa-9a2a-a1ca6e194565 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:40.010138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52324442-5fdb-49fa-9a2a-a1ca6e194565 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:40.010430 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52324442-5fdb-49fa-9a2a-a1ca6e194565 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:40.042690 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-52324442-5fdb-49fa-9a2a-a1ca6e194565 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:19:40.045077 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-52324442-5fdb-49fa-9a2a-a1ca6e194565 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:19:40.106690 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:40.164241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 642/1294] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:19:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 14:19:40.451044 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1df969-ef71-4508-aed6-92dd697221ba None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:40.485798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:40.568853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1df969-ef71-4508-aed6-92dd697221ba None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:40.604687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9df73b5-c56e-4964-943e-1721a2333563 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:40.605299 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9df73b5-c56e-4964-943e-1721a2333563 None admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:40.605577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9df73b5-c56e-4964-943e-1721a2333563 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:40.605850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9df73b5-c56e-4964-943e-1721a2333563 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:40.606516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9df73b5-c56e-4964-943e-1721a2333563 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:40.607882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9df73b5-c56e-4964-943e-1721a2333563 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:40.623842 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 653/1295] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Oct 4 14:19:40 2022] POST /identity/v3/services => generated 246 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:19:42.932380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9479c1-7683-4da0-a6d7-b85cfa597b5a None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:42.932956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9479c1-7683-4da0-a6d7-b85cfa597b5a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:42.932956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9479c1-7683-4da0-a6d7-b85cfa597b5a None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:42.934342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 643/1296] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:19:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:19:42.941179 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6ed8dd-9a05-4afa-8555-3f8afbc88c4a None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:42.941705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6ed8dd-9a05-4afa-8555-3f8afbc88c4a None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:42.941705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6ed8dd-9a05-4afa-8555-3f8afbc88c4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:42.961247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-9d6ed8dd-9a05-4afa-8555-3f8afbc88c4a None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:19:42.962466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9d6ed8dd-9a05-4afa-8555-3f8afbc88c4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:19:43.049978 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 654/1297] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:19:42 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:19:43.335846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9479c1-7683-4da0-a6d7-b85cfa597b5a None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:43.456674 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9479c1-7683-4da0-a6d7-b85cfa597b5a None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:43.501364 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb54c9a-d7f1-488d-919a-4fefd41c5f05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:43.502826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb54c9a-d7f1-488d-919a-4fefd41c5f05 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:43.503301 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb54c9a-d7f1-488d-919a-4fefd41c5f05 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:43.503775 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb54c9a-d7f1-488d-919a-4fefd41c5f05 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:43.504844 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb54c9a-d7f1-488d-919a-4fefd41c5f05 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:43.507333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb54c9a-d7f1-488d-919a-4fefd41c5f05 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:43.523458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-8eb54c9a-d7f1-488d-919a-4fefd41c5f05 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 14:19:43.523458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:43.523458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:19:43.523458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:19:43.523458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:19:43.523458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:19:43.523458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:19:43.523458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:19:43.523458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 14:19:43.523458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:43.523458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:19:43.523458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:43.523458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:43.523458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:43.523458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:43.523458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:43.523458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:43.525074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:43.525074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:43.525074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:43.525074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:43.525074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:43.525074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:43.525074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:19:43.525074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:19:43.525074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:19:43.525074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:19:43.525074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:43.525074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:43.525074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:43.525074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:43.525074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:43.525074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:43.529146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:43.529146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:43.529146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:43.529146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:43.529146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:43.529146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:43.529146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:43.529146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:43.529146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:43.529146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:43.529146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:19:43.529146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:19:43.529146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 14:19:43.529146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:43.529146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 644/1298] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Oct 4 14:19:43 2022] GET /identity/v3/services/placement => generated 90 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:43.531944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6ed8dd-9a05-4afa-8555-3f8afbc88c4a None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:43.553862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:43.650425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6ed8dd-9a05-4afa-8555-3f8afbc88c4a None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:43.692392 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01913c9e-dbe3-41e5-a601-f94da3c105ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:43.693011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01913c9e-dbe3-41e5-a601-f94da3c105ac None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:43.693286 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01913c9e-dbe3-41e5-a601-f94da3c105ac None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:43.693557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01913c9e-dbe3-41e5-a601-f94da3c105ac None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:43.694212 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01913c9e-dbe3-41e5-a601-f94da3c105ac None admin] RBAC: Adding query filter params (name=placement) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:43.694511 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01913c9e-dbe3-41e5-a601-f94da3c105ac None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:43.695914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01913c9e-dbe3-41e5-a601-f94da3c105ac None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:43.702045 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 655/1299] 10.222.0.64 () {56 vars in 1116 bytes} [Tue Oct 4 14:19:43 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:19:43.708761 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb54c9a-d7f1-488d-919a-4fefd41c5f05 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:43.744963 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:43.829599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb54c9a-d7f1-488d-919a-4fefd41c5f05 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:43.871819 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb58f65-6511-4f2d-8d00-954bb4a39475 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:43.872476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb58f65-6511-4f2d-8d00-954bb4a39475 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:43.872759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb58f65-6511-4f2d-8d00-954bb4a39475 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:43.873035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb58f65-6511-4f2d-8d00-954bb4a39475 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:43.873754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb58f65-6511-4f2d-8d00-954bb4a39475 None admin] RBAC: Adding query filter params (interface=public, service_id=372be2f667144d17b2985ceac6952aac, region_id=RegionOne) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:43.874064 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb58f65-6511-4f2d-8d00-954bb4a39475 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:43.875471 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb58f65-6511-4f2d-8d00-954bb4a39475 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:43.885111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 645/1300] 10.222.0.64 () {56 vars in 1251 bytes} [Tue Oct 4 14:19:43 2022] GET /identity/v3/endpoints?service_id=372be2f667144d17b2985ceac6952aac&interface=public®ion_id=RegionOne => generated 194 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:43.891508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01913c9e-dbe3-41e5-a601-f94da3c105ac None admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:44.006619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:44.042803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01913c9e-dbe3-41e5-a601-f94da3c105ac None admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:44.098279 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e303c66-af5b-4a83-8666-241c97a232b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:44.098901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e303c66-af5b-4a83-8666-241c97a232b6 None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:44.099177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e303c66-af5b-4a83-8666-241c97a232b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:44.099828 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e303c66-af5b-4a83-8666-241c97a232b6 None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:44.100469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e303c66-af5b-4a83-8666-241c97a232b6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:44.101466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e303c66-af5b-4a83-8666-241c97a232b6 None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:44.109513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 656/1301] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Oct 4 14:19:43 2022] GET /identity/v3/services => generated 2191 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:19:46.411254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb365667-3987-429e-9cf6-04594a07dde2 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:46.411904 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb365667-3987-429e-9cf6-04594a07dde2 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:46.412218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb365667-3987-429e-9cf6-04594a07dde2 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:46.414752 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 646/1302] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:19:46 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:19:46.420972 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e95106b-3921-4a6f-aec7-9dcbea3198a8 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:46.421790 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e95106b-3921-4a6f-aec7-9dcbea3198a8 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:46.421990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e95106b-3921-4a6f-aec7-9dcbea3198a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:46.443395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-6e95106b-3921-4a6f-aec7-9dcbea3198a8 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:19:46.444735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6e95106b-3921-4a6f-aec7-9dcbea3198a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:19:46.527073 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 657/1303] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:19:46 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 14:19:46.808343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb365667-3987-429e-9cf6-04594a07dde2 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:46.920230 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:46.929108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb365667-3987-429e-9cf6-04594a07dde2 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:46.974769 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ccf3ae-e673-4588-ab02-2926a3cb3801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:46.975420 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ccf3ae-e673-4588-ab02-2926a3cb3801 None admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:46.975702 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ccf3ae-e673-4588-ab02-2926a3cb3801 None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:46.975975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ccf3ae-e673-4588-ab02-2926a3cb3801 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:46.976593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ccf3ae-e673-4588-ab02-2926a3cb3801 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:46.977986 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ccf3ae-e673-4588-ab02-2926a3cb3801 None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:46.987784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application [None req-66ccf3ae-e673-4588-ab02-2926a3cb3801 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 14:19:46.987784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:46.987784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 14:19:46.987784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 14:19:46.987784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 14:19:46.987784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 14:19:46.987784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 14:19:46.987784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:19:46.987784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 14:19:46.987784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:46.987784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 14:19:46.987784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:46.987784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 14:19:46.987784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 14:19:46.987784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 14:19:46.987784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 14:19:46.987784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 14:19:46.989751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 14:19:46.989751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 14:19:46.989751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 14:19:46.989751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 14:19:46.989751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 14:19:46.989751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 14:19:46.989751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 14:19:46.989751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 14:19:46.989751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 14:19:46.989751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 14:19:46.989751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 14:19:46.989751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 14:19:46.989751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 14:19:46.989751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 14:19:46.989751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 14:19:46.989751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 14:19:46.991409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 14:19:46.991409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application with Lock( Oct 04 14:19:46.991409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 14:19:46.991409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self._enter() Oct 04 14:19:46.991409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 14:19:46.991409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 14:19:46.991409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 14:19:46.991409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application return self.creator() Oct 04 14:19:46.991409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 14:19:46.991409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application created_value = creator( Oct 04 14:19:46.991409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 14:19:46.991409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 14:19:46.991409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 14:19:46.991409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: ERROR keystone.server.flask.application Oct 04 14:19:46.991409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 647/1304] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Oct 4 14:19:46 2022] GET /identity/v3/services/placement => generated 90 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 14:19:46.995823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e95106b-3921-4a6f-aec7-9dcbea3198a8 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:47.092604 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:47.130907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e95106b-3921-4a6f-aec7-9dcbea3198a8 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:47.168540 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59766f3e-a7aa-45a2-b3d3-f0b9077dd84e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:47.169176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59766f3e-a7aa-45a2-b3d3-f0b9077dd84e None admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:47.169458 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59766f3e-a7aa-45a2-b3d3-f0b9077dd84e None admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:47.169819 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59766f3e-a7aa-45a2-b3d3-f0b9077dd84e None admin] PATH_INFO: `/v3/services` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:47.170488 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59766f3e-a7aa-45a2-b3d3-f0b9077dd84e None admin] RBAC: Adding query filter params (name=placement) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:19:47.170796 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59766f3e-a7aa-45a2-b3d3-f0b9077dd84e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:47.172292 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59766f3e-a7aa-45a2-b3d3-f0b9077dd84e None admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:47.178766 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 658/1305] 10.222.0.64 () {56 vars in 1116 bytes} [Tue Oct 4 14:19:46 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:19:47.187873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ccf3ae-e673-4588-ab02-2926a3cb3801 None admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:19:47.307505 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ccf3ae-e673-4588-ab02-2926a3cb3801 None admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:19:47.355474 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332ac42d-1b27-4eb5-abdb-ed41caf0428a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:19:47.356011 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332ac42d-1b27-4eb5-abdb-ed41caf0428a None admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:19:47.356299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332ac42d-1b27-4eb5-abdb-ed41caf0428a None admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:19:47.356635 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332ac42d-1b27-4eb5-abdb-ed41caf0428a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:19:47.357479 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332ac42d-1b27-4eb5-abdb-ed41caf0428a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:19:47.358868 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332ac42d-1b27-4eb5-abdb-ed41caf0428a None admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:19:47.382191 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 648/1306] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:19:47 2022] POST /identity/v3/endpoints => generated 331 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:19:57.178604 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=30 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:19:57.377445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:00.862548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982b372b-9c82-4ef0-94de-cfdc22c49e06 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:00.863411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982b372b-9c82-4ef0-94de-cfdc22c49e06 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:00.863411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982b372b-9c82-4ef0-94de-cfdc22c49e06 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:00.868621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 659/1307] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:21:00 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:21:00.870235 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8e9fa7-81cb-47f2-8e04-e45aee027ac6 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:00.870883 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8e9fa7-81cb-47f2-8e04-e45aee027ac6 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:00.870883 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8e9fa7-81cb-47f2-8e04-e45aee027ac6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:01.045514 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-5e8e9fa7-81cb-47f2-8e04-e45aee027ac6 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:21:01.046958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-5e8e9fa7-81cb-47f2-8e04-e45aee027ac6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:01.307859 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 649/1308] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:21:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 439 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:21:01.898946 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efdf319-405c-44c3-baa1-aa95312841cd None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:01.899754 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efdf319-405c-44c3-baa1-aa95312841cd None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:01.899754 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efdf319-405c-44c3-baa1-aa95312841cd None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:01.900850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 660/1309] 10.222.0.64 () {54 vars in 964 bytes} [Tue Oct 4 14:21:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:21:01.910832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c8cf91-8f71-4951-a5a6-2d4f053d13c3 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:01.912292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c8cf91-8f71-4951-a5a6-2d4f053d13c3 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:01.913105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c8cf91-8f71-4951-a5a6-2d4f053d13c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:02.164670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-32c8cf91-8f71-4951-a5a6-2d4f053d13c3 None None] MFA Rules not processed for user `8dbeccfb9a194992959f0b341ef0d4a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:21:02.170391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-32c8cf91-8f71-4951-a5a6-2d4f053d13c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:02.247903 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:02.371399 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 650/1310] 10.222.0.64 () {58 vars in 1061 bytes} [Tue Oct 4 14:21:01 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 462 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:21:02.381654 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efdf319-405c-44c3-baa1-aa95312841cd None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:21:02.383765 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7efdf319-405c-44c3-baa1-aa95312841cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:02.531382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efdf319-405c-44c3-baa1-aa95312841cd None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:21:02.629470 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907c3595-d287-423d-acf6-262af7cd876c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:21:02.630251 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907c3595-d287-423d-acf6-262af7cd876c service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:02.630644 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907c3595-d287-423d-acf6-262af7cd876c service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:02.631026 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907c3595-d287-423d-acf6-262af7cd876c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:02.632851 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-907c3595-d287-423d-acf6-262af7cd876c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:02.726294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:02.812142 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907c3595-d287-423d-acf6-262af7cd876c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:21:02.813719 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907c3595-d287-423d-acf6-262af7cd876c service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:21:02.837329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 661/1311] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:21:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 456 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:21:05.877911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd18a1-63bf-415f-9c54-c3441cb1318e None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:05.878640 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd18a1-63bf-415f-9c54-c3441cb1318e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:05.878640 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd18a1-63bf-415f-9c54-c3441cb1318e None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:05.879795 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 651/1312] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:21:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:21:05.886614 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb0cade-cd21-4d02-bafe-f995bfd6f838 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:05.887291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb0cade-cd21-4d02-bafe-f995bfd6f838 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:05.887291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb0cade-cd21-4d02-bafe-f995bfd6f838 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:06.000134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-0fb0cade-cd21-4d02-bafe-f995bfd6f838 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:21:06.001385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-0fb0cade-cd21-4d02-bafe-f995bfd6f838 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:06.107675 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 662/1313] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:21:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:21:06.396600 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addd18a1-63bf-415f-9c54-c3441cb1318e None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:21:06.398208 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-addd18a1-63bf-415f-9c54-c3441cb1318e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:06.528816 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addd18a1-63bf-415f-9c54-c3441cb1318e None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:21:06.622016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:06.625595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dfd9d8-23ab-46ac-ad18-000d79097dff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:21:06.626579 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dfd9d8-23ab-46ac-ad18-000d79097dff service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:06.628288 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dfd9d8-23ab-46ac-ad18-000d79097dff service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:06.628660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dfd9d8-23ab-46ac-ad18-000d79097dff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:06.631038 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-38dfd9d8-23ab-46ac-ad18-000d79097dff service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:06.759449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dfd9d8-23ab-46ac-ad18-000d79097dff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:21:06.762125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dfd9d8-23ab-46ac-ad18-000d79097dff service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:21:06.783243 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 652/1314] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:21:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:21:09.390552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc62a016-d539-41b2-b266-6b7266432a30 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:09.391171 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc62a016-d539-41b2-b266-6b7266432a30 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:09.391171 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc62a016-d539-41b2-b266-6b7266432a30 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:09.393434 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 663/1315] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:21:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:21:09.401014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b20edb-f9be-4f49-b9af-8c94cc481503 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:09.401575 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b20edb-f9be-4f49-b9af-8c94cc481503 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:09.401575 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b20edb-f9be-4f49-b9af-8c94cc481503 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:09.489204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-83b20edb-f9be-4f49-b9af-8c94cc481503 None None] MFA Rules not processed for user `ac210aede44f454dbb9c3aa4a942ffab`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:21:09.490471 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-83b20edb-f9be-4f49-b9af-8c94cc481503 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:09.552114 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:09.601764 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 653/1316] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:21:09 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:21:09.908795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc62a016-d539-41b2-b266-6b7266432a30 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:21:09.910904 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-fc62a016-d539-41b2-b266-6b7266432a30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:09.923834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:10.030597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc62a016-d539-41b2-b266-6b7266432a30 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:21:10.120364 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee84f6b-5472-4654-81b6-f65ef7dfb810 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:21:10.121019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee84f6b-5472-4654-81b6-f65ef7dfb810 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:10.121300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee84f6b-5472-4654-81b6-f65ef7dfb810 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:10.121576 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee84f6b-5472-4654-81b6-f65ef7dfb810 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:10.123276 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1ee84f6b-5472-4654-81b6-f65ef7dfb810 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:10.200124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:10.257281 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee84f6b-5472-4654-81b6-f65ef7dfb810 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:21:10.258735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee84f6b-5472-4654-81b6-f65ef7dfb810 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:21:10.281064 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 664/1317] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:21:09 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:21:13.165614 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d9916e-f6b0-4b89-8194-2168e59c1cef None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:13.166154 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d9916e-f6b0-4b89-8194-2168e59c1cef None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:13.166272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d9916e-f6b0-4b89-8194-2168e59c1cef None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:13.167519 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 654/1318] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:21:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:21:13.172995 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e04843-44f5-4285-b588-342e63b7fe34 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:13.173405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e04843-44f5-4285-b588-342e63b7fe34 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:13.173537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e04843-44f5-4285-b588-342e63b7fe34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:13.242968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-91e04843-44f5-4285-b588-342e63b7fe34 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:21:13.244430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-91e04843-44f5-4285-b588-342e63b7fe34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:13.349655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 665/1319] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:21:13 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:21:13.640816 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d9916e-f6b0-4b89-8194-2168e59c1cef None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:21:13.642362 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-17d9916e-f6b0-4b89-8194-2168e59c1cef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:13.750946 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d9916e-f6b0-4b89-8194-2168e59c1cef None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:21:13.828305 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a4883b-75dd-4150-8c62-0f1ba6827aec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:21:13.828936 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a4883b-75dd-4150-8c62-0f1ba6827aec service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:13.829214 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a4883b-75dd-4150-8c62-0f1ba6827aec service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:13.829502 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a4883b-75dd-4150-8c62-0f1ba6827aec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:13.832223 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-b6a4883b-75dd-4150-8c62-0f1ba6827aec service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:13.842387 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:13.970848 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a4883b-75dd-4150-8c62-0f1ba6827aec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:21:13.972205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a4883b-75dd-4150-8c62-0f1ba6827aec service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:21:13.994929 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 655/1320] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:21:13 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:21:17.093160 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d21dab3-3561-46da-8f38-0eef61066cf7 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:17.093741 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d21dab3-3561-46da-8f38-0eef61066cf7 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:17.093741 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d21dab3-3561-46da-8f38-0eef61066cf7 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:17.095185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 666/1321] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:21:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:21:17.100776 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06440e39-4060-43c2-a038-9429d43f54d4 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:17.101243 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06440e39-4060-43c2-a038-9429d43f54d4 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:17.101343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06440e39-4060-43c2-a038-9429d43f54d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:17.171844 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-06440e39-4060-43c2-a038-9429d43f54d4 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:21:17.173090 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-06440e39-4060-43c2-a038-9429d43f54d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:17.241342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:17.317182 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 656/1322] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:21:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:21:17.620518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d21dab3-3561-46da-8f38-0eef61066cf7 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:21:17.622451 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-8d21dab3-3561-46da-8f38-0eef61066cf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:17.648285 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:17.778109 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d21dab3-3561-46da-8f38-0eef61066cf7 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:21:17.929035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2870c84a-5e1f-428d-96df-ef32300b6b68 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:21:17.930053 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2870c84a-5e1f-428d-96df-ef32300b6b68 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:17.930524 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2870c84a-5e1f-428d-96df-ef32300b6b68 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:17.931003 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2870c84a-5e1f-428d-96df-ef32300b6b68 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:17.934006 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-2870c84a-5e1f-428d-96df-ef32300b6b68 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:18.044129 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:18.098589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2870c84a-5e1f-428d-96df-ef32300b6b68 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:21:18.100581 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2870c84a-5e1f-428d-96df-ef32300b6b68 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:21:18.123112 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 667/1323] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:21:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 503 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:21:20.561715 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fa066e-d7ef-4647-8bcd-5b106c5d0999 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:20.562255 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fa066e-d7ef-4647-8bcd-5b106c5d0999 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:20.562255 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fa066e-d7ef-4647-8bcd-5b106c5d0999 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:20.563620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 657/1324] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:21:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:21:20.569725 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5b202a-92ac-40d2-ba53-3c06073b45eb None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:20.570288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5b202a-92ac-40d2-ba53-3c06073b45eb None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:20.570288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5b202a-92ac-40d2-ba53-3c06073b45eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:20.668764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-bd5b202a-92ac-40d2-ba53-3c06073b45eb None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:21:20.670380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-bd5b202a-92ac-40d2-ba53-3c06073b45eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:20.813637 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 668/1325] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:21:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:21:21.173645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95905a0-b852-4b71-82fb-205f399f5f2f None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:21.174009 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95905a0-b852-4b71-82fb-205f399f5f2f None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:21.174225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95905a0-b852-4b71-82fb-205f399f5f2f None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:21.175922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 658/1326] 10.222.0.64 () {54 vars in 964 bytes} [Tue Oct 4 14:21:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:21:21.183638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9956ef8a-137b-4188-8cea-9c978e54d9ba None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:21.184078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9956ef8a-137b-4188-8cea-9c978e54d9ba None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:21.184545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9956ef8a-137b-4188-8cea-9c978e54d9ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:21.242378 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:21.300005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-9956ef8a-137b-4188-8cea-9c978e54d9ba None None] MFA Rules not processed for user `8dbeccfb9a194992959f0b341ef0d4a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:21:21.301176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9956ef8a-137b-4188-8cea-9c978e54d9ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:21.388582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 669/1327] 10.222.0.64 () {58 vars in 1061 bytes} [Tue Oct 4 14:21:21 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:21:21.398004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95905a0-b852-4b71-82fb-205f399f5f2f None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:21:21.399531 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a95905a0-b852-4b71-82fb-205f399f5f2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:21.522046 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95905a0-b852-4b71-82fb-205f399f5f2f None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:21:21.607202 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7afc5ce-be12-4b15-b4c9-1ba634c74f2c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:21:21.608207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7afc5ce-be12-4b15-b4c9-1ba634c74f2c service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:21.608207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7afc5ce-be12-4b15-b4c9-1ba634c74f2c service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:21.608612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7afc5ce-be12-4b15-b4c9-1ba634c74f2c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:21.610007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d7afc5ce-be12-4b15-b4c9-1ba634c74f2c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:21.629611 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:21.733692 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7afc5ce-be12-4b15-b4c9-1ba634c74f2c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:21:21.735066 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7afc5ce-be12-4b15-b4c9-1ba634c74f2c service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:21:21.756293 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 659/1328] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:21:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:21:24.806977 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa6bf51-7411-4771-b3ac-716c1d3db669 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:24.807550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa6bf51-7411-4771-b3ac-716c1d3db669 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:24.807550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa6bf51-7411-4771-b3ac-716c1d3db669 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:24.808994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 670/1329] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:21:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:21:24.815129 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bedf714-f576-4042-8db4-936f1147e21d None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:24.815744 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bedf714-f576-4042-8db4-936f1147e21d None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:24.815837 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bedf714-f576-4042-8db4-936f1147e21d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:24.897621 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-4bedf714-f576-4042-8db4-936f1147e21d None None] MFA Rules not processed for user `ac210aede44f454dbb9c3aa4a942ffab`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:21:24.898754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-4bedf714-f576-4042-8db4-936f1147e21d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:24.981996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:25.043949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 660/1330] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:21:24 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:21:25.387670 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa6bf51-7411-4771-b3ac-716c1d3db669 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:21:25.389469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-4fa6bf51-7411-4771-b3ac-716c1d3db669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:25.566025 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa6bf51-7411-4771-b3ac-716c1d3db669 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:21:25.611012 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:25.704054 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b629fdab-39fd-4db2-aafe-258c3b1879ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:21:25.704598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b629fdab-39fd-4db2-aafe-258c3b1879ef service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:25.704868 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b629fdab-39fd-4db2-aafe-258c3b1879ef service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:25.705474 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b629fdab-39fd-4db2-aafe-258c3b1879ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:25.706827 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b629fdab-39fd-4db2-aafe-258c3b1879ef service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:25.843612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b629fdab-39fd-4db2-aafe-258c3b1879ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:21:25.845165 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b629fdab-39fd-4db2-aafe-258c3b1879ef service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:21:25.867652 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 671/1331] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:21:25 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 480 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:21:28.851278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e731308c-c00e-4600-83cf-669ef33639e1 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:28.851823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e731308c-c00e-4600-83cf-669ef33639e1 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:28.851823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e731308c-c00e-4600-83cf-669ef33639e1 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:28.853196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 661/1332] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:21:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:21:28.858847 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a052cb-6c20-4fda-8524-7d2daeccde88 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:28.859335 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a052cb-6c20-4fda-8524-7d2daeccde88 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:28.859432 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a052cb-6c20-4fda-8524-7d2daeccde88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:28.928148 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:28.932052 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-15a052cb-6c20-4fda-8524-7d2daeccde88 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:21:28.933699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-15a052cb-6c20-4fda-8524-7d2daeccde88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:29.045849 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 672/1333] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:21:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:21:29.353487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e731308c-c00e-4600-83cf-669ef33639e1 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:21:29.355057 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-e731308c-c00e-4600-83cf-669ef33639e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:29.493079 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e731308c-c00e-4600-83cf-669ef33639e1 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:21:29.581166 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d63a91-837a-476c-a7aa-0ea69abe20c2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:21:29.581784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d63a91-837a-476c-a7aa-0ea69abe20c2 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:29.582064 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d63a91-837a-476c-a7aa-0ea69abe20c2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:29.582334 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d63a91-837a-476c-a7aa-0ea69abe20c2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:29.583982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c5d63a91-837a-476c-a7aa-0ea69abe20c2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:29.618291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:29.713798 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d63a91-837a-476c-a7aa-0ea69abe20c2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:21:29.715288 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d63a91-837a-476c-a7aa-0ea69abe20c2 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:21:29.735959 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 662/1334] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:21:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:21:32.074586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a268692-e386-4237-ad06-5b63435a66b0 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:32.075428 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a268692-e386-4237-ad06-5b63435a66b0 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:32.075428 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a268692-e386-4237-ad06-5b63435a66b0 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:32.076863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 673/1335] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:21:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:21:32.083381 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cd3f8c-c5b3-48ff-bda8-508fecb55665 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:32.083922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cd3f8c-c5b3-48ff-bda8-508fecb55665 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:32.084035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cd3f8c-c5b3-48ff-bda8-508fecb55665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:32.167025 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-76cd3f8c-c5b3-48ff-bda8-508fecb55665 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:21:32.169340 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-76cd3f8c-c5b3-48ff-bda8-508fecb55665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:32.281522 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:32.308977 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 663/1336] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:21:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:21:32.612994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a268692-e386-4237-ad06-5b63435a66b0 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:21:32.614528 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7a268692-e386-4237-ad06-5b63435a66b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:32.732782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a268692-e386-4237-ad06-5b63435a66b0 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:21:32.780711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:32.832336 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6ca781-9895-4ab8-9253-179a33b0daf3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:21:32.832951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ca781-9895-4ab8-9253-179a33b0daf3 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:32.833230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ca781-9895-4ab8-9253-179a33b0daf3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:32.833505 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ca781-9895-4ab8-9253-179a33b0daf3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:32.835319 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-4c6ca781-9895-4ab8-9253-179a33b0daf3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:32.974316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6ca781-9895-4ab8-9253-179a33b0daf3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:21:32.975761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6ca781-9895-4ab8-9253-179a33b0daf3 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:21:32.995275 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 674/1337] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:21:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:21:35.763041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a76e9d-4f5e-4b32-95a5-a7b4c8e58f75 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:35.763602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a76e9d-4f5e-4b32-95a5-a7b4c8e58f75 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:35.763602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a76e9d-4f5e-4b32-95a5-a7b4c8e58f75 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:35.765644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 664/1338] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:21:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:21:35.775518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2920aae0-7ace-48d3-8ecd-b7ec07533e12 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:35.776403 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2920aae0-7ace-48d3-8ecd-b7ec07533e12 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:35.776403 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2920aae0-7ace-48d3-8ecd-b7ec07533e12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:35.856790 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-2920aae0-7ace-48d3-8ecd-b7ec07533e12 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:21:35.858777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-2920aae0-7ace-48d3-8ecd-b7ec07533e12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:35.877625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:36.020638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 675/1339] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:21:35 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:21:36.470550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a76e9d-4f5e-4b32-95a5-a7b4c8e58f75 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:21:36.473284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-14a76e9d-4f5e-4b32-95a5-a7b4c8e58f75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:36.616123 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a76e9d-4f5e-4b32-95a5-a7b4c8e58f75 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:21:36.723359 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98a1c16-758f-4198-a074-d9a352e4613f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:21:36.724127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98a1c16-758f-4198-a074-d9a352e4613f service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:36.724559 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98a1c16-758f-4198-a074-d9a352e4613f service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:36.724944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98a1c16-758f-4198-a074-d9a352e4613f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:36.727553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a98a1c16-758f-4198-a074-d9a352e4613f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:36.780993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:36.874447 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98a1c16-758f-4198-a074-d9a352e4613f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:21:36.876044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98a1c16-758f-4198-a074-d9a352e4613f service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:21:36.898978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 665/1340] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:21:36 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:21:41.704172 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875915bc-2afd-498c-adf9-707411fa151b None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:41.704834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875915bc-2afd-498c-adf9-707411fa151b None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:41.704834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875915bc-2afd-498c-adf9-707411fa151b None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:41.707151 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 676/1341] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:21:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:21:41.713364 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c5df8a-9b3f-4271-bb92-1869f61afdb1 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:41.713917 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c5df8a-9b3f-4271-bb92-1869f61afdb1 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:41.714058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c5df8a-9b3f-4271-bb92-1869f61afdb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:41.794479 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-20c5df8a-9b3f-4271-bb92-1869f61afdb1 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:21:41.795928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-20c5df8a-9b3f-4271-bb92-1869f61afdb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:41.906352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:41.920628 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 666/1342] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:21:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:21:42.220273 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875915bc-2afd-498c-adf9-707411fa151b None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:21:42.222468 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-875915bc-2afd-498c-adf9-707411fa151b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:42.331694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875915bc-2afd-498c-adf9-707411fa151b None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:21:42.373437 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:42.417542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9688c0a7-2db8-4ffb-9130-ebfcc4c634e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:21:42.418159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9688c0a7-2db8-4ffb-9130-ebfcc4c634e3 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:42.418433 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9688c0a7-2db8-4ffb-9130-ebfcc4c634e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:42.418700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9688c0a7-2db8-4ffb-9130-ebfcc4c634e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:42.420387 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9688c0a7-2db8-4ffb-9130-ebfcc4c634e3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:42.539176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9688c0a7-2db8-4ffb-9130-ebfcc4c634e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:21:42.540588 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9688c0a7-2db8-4ffb-9130-ebfcc4c634e3 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:21:42.561974 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 677/1343] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:21:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:21:45.520687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aed2fb-7f53-4c37-a910-6b9200c1bd83 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:45.521237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aed2fb-7f53-4c37-a910-6b9200c1bd83 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:45.521237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aed2fb-7f53-4c37-a910-6b9200c1bd83 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:45.522732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 667/1344] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:21:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:21:45.529444 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da523251-1a96-4fb0-a56c-cd4e9cb32616 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:45.530110 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da523251-1a96-4fb0-a56c-cd4e9cb32616 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:45.530110 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da523251-1a96-4fb0-a56c-cd4e9cb32616 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:45.638105 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-da523251-1a96-4fb0-a56c-cd4e9cb32616 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:21:45.640038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-da523251-1a96-4fb0-a56c-cd4e9cb32616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:45.678486 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:45.780072 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 678/1345] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:21:45 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:21:46.069365 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10aed2fb-7f53-4c37-a910-6b9200c1bd83 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:21:46.070916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-10aed2fb-7f53-4c37-a910-6b9200c1bd83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:46.222570 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10aed2fb-7f53-4c37-a910-6b9200c1bd83 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:21:46.327551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60404cf5-a740-4539-91c4-d83a6f43bd26 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:21:46.328066 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60404cf5-a740-4539-91c4-d83a6f43bd26 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:46.328373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60404cf5-a740-4539-91c4-d83a6f43bd26 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:46.328645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60404cf5-a740-4539-91c4-d83a6f43bd26 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:46.330660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-60404cf5-a740-4539-91c4-d83a6f43bd26 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:46.391705 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:46.487598 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60404cf5-a740-4539-91c4-d83a6f43bd26 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:21:46.489134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60404cf5-a740-4539-91c4-d83a6f43bd26 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:21:46.510749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 668/1346] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:21:46 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:21:51.218782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f1d3eb-6180-42c2-9c03-a05d6a634e67 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:51.219360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f1d3eb-6180-42c2-9c03-a05d6a634e67 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:51.219740 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f1d3eb-6180-42c2-9c03-a05d6a634e67 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:51.224374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 679/1347] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:21:51 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:21:51.231617 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c31c25d-6665-4076-88db-0146e9792354 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:51.231617 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c31c25d-6665-4076-88db-0146e9792354 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:51.232246 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c31c25d-6665-4076-88db-0146e9792354 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:51.342540 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-9c31c25d-6665-4076-88db-0146e9792354 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:21:51.343892 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9c31c25d-6665-4076-88db-0146e9792354 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:51.481104 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:51.483404 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 669/1348] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:21:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:21:51.782066 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f1d3eb-6180-42c2-9c03-a05d6a634e67 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:21:51.784229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a5f1d3eb-6180-42c2-9c03-a05d6a634e67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:51.910899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f1d3eb-6180-42c2-9c03-a05d6a634e67 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:21:51.965170 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:51.994179 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03dfb03-1d5a-4649-b655-229d2f20958b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:21:51.994798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03dfb03-1d5a-4649-b655-229d2f20958b service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:51.995067 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03dfb03-1d5a-4649-b655-229d2f20958b service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:51.995340 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03dfb03-1d5a-4649-b655-229d2f20958b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:51.997076 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b03dfb03-1d5a-4649-b655-229d2f20958b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:52.126198 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03dfb03-1d5a-4649-b655-229d2f20958b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:21:52.128520 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03dfb03-1d5a-4649-b655-229d2f20958b service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:21:52.150312 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 680/1349] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:21:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:21:57.782890 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c789dab-67f9-4a5e-8ce5-bd1f2bf5abb1 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:57.784710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c789dab-67f9-4a5e-8ce5-bd1f2bf5abb1 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:57.784710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c789dab-67f9-4a5e-8ce5-bd1f2bf5abb1 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:57.788351 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 670/1350] 10.222.0.64 () {54 vars in 921 bytes} [Tue Oct 4 14:21:57 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:21:57.797635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6f4e69-072f-46a3-a995-b5dcc855ce0c None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:21:57.798539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6f4e69-072f-46a3-a995-b5dcc855ce0c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:21:57.799427 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6f4e69-072f-46a3-a995-b5dcc855ce0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:21:57.933600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-bf6f4e69-072f-46a3-a995-b5dcc855ce0c None None] MFA Rules not processed for user `436deb3d60c54a799700986519b2ef68`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:21:57.936859 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-bf6f4e69-072f-46a3-a995-b5dcc855ce0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:21:58.003902 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:21:58.164766 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 681/1351] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Oct 4 14:21:57 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 369 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:22:04.069814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e82f0b7-53d7-4762-bda8-69ae04a488c6 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:04.070376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e82f0b7-53d7-4762-bda8-69ae04a488c6 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:04.070376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e82f0b7-53d7-4762-bda8-69ae04a488c6 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:04.071633 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 671/1352] 10.222.0.64 () {54 vars in 921 bytes} [Tue Oct 4 14:22:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:22:04.079769 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795b5f60-e8f0-4733-a310-1424dd260e45 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:04.081970 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795b5f60-e8f0-4733-a310-1424dd260e45 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:04.083814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795b5f60-e8f0-4733-a310-1424dd260e45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:04.259756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-795b5f60-e8f0-4733-a310-1424dd260e45 None None] MFA Rules not processed for user `436deb3d60c54a799700986519b2ef68`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:04.261669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-795b5f60-e8f0-4733-a310-1424dd260e45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:04.415113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:04.492640 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 682/1353] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Oct 4 14:22:04 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 416 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:22:05.672715 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4359354d-3b66-4f67-9f90-de1f123af953 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:05.673261 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4359354d-3b66-4f67-9f90-de1f123af953 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:05.673566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4359354d-3b66-4f67-9f90-de1f123af953 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:05.676350 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 672/1354] 10.222.0.64 () {54 vars in 921 bytes} [Tue Oct 4 14:22:05 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:22:05.684798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9af4acb-8dbf-4334-9a7e-ffef97f96656 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:05.685359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9af4acb-8dbf-4334-9a7e-ffef97f96656 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:05.685718 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9af4acb-8dbf-4334-9a7e-ffef97f96656 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:05.793270 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-a9af4acb-8dbf-4334-9a7e-ffef97f96656 None None] MFA Rules not processed for user `436deb3d60c54a799700986519b2ef68`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:05.794486 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a9af4acb-8dbf-4334-9a7e-ffef97f96656 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:05.938393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 683/1355] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Oct 4 14:22:05 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:22:09.695756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18df2356-45bd-40a4-a9cc-c910f9f829a2 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:09.696627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18df2356-45bd-40a4-a9cc-c910f9f829a2 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:09.696627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18df2356-45bd-40a4-a9cc-c910f9f829a2 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:09.698059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 673/1356] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:22:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:22:09.704991 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c188f9e6-6ddb-4cb3-bb13-8b674fe77dca None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:09.706665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c188f9e6-6ddb-4cb3-bb13-8b674fe77dca None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:09.707182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c188f9e6-6ddb-4cb3-bb13-8b674fe77dca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:09.776687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:09.820115 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-c188f9e6-6ddb-4cb3-bb13-8b674fe77dca None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:09.821417 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c188f9e6-6ddb-4cb3-bb13-8b674fe77dca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:09.996416 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 684/1357] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:22:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 293 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:22:10.485335 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9612b3ea-7333-463c-92fd-891c72e2a866 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:10.485782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9612b3ea-7333-463c-92fd-891c72e2a866 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:10.486232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9612b3ea-7333-463c-92fd-891c72e2a866 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:10.489366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 674/1358] 10.222.0.64 () {54 vars in 961 bytes} [Tue Oct 4 14:22:10 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:22:10.501503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dc4efe-69e1-40d3-bf82-1c856dd7a567 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:10.502005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dc4efe-69e1-40d3-bf82-1c856dd7a567 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:10.503814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dc4efe-69e1-40d3-bf82-1c856dd7a567 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:10.616183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-69dc4efe-69e1-40d3-bf82-1c856dd7a567 None None] MFA Rules not processed for user `03081d7d581745c88c12e9d089fcb2d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:10.618421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-69dc4efe-69e1-40d3-bf82-1c856dd7a567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:10.699719 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:10.801994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 685/1359] 10.222.0.64 () {58 vars in 1058 bytes} [Tue Oct 4 14:22:10 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:22:10.810678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9612b3ea-7333-463c-92fd-891c72e2a866 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:22:10.816057 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9612b3ea-7333-463c-92fd-891c72e2a866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:11.138051 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9612b3ea-7333-463c-92fd-891c72e2a866 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:22:11.303672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:11.368973 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754d0d42-abcb-4365-b609-8f6f17726843 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:22:11.369826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754d0d42-abcb-4365-b609-8f6f17726843 service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:11.370234 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754d0d42-abcb-4365-b609-8f6f17726843 service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:11.370613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754d0d42-abcb-4365-b609-8f6f17726843 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:11.372502 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-754d0d42-abcb-4365-b609-8f6f17726843 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:11.502986 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754d0d42-abcb-4365-b609-8f6f17726843 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:22:11.504843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754d0d42-abcb-4365-b609-8f6f17726843 service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:22:11.530839 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 675/1360] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:22:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 721 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:22:14.640677 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe40229-9ba5-457e-8928-0698ede2c388 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:14.641526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe40229-9ba5-457e-8928-0698ede2c388 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:14.641526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe40229-9ba5-457e-8928-0698ede2c388 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:14.642751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 686/1361] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:22:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:22:14.650781 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb713886-f2f0-4cda-a759-d8c0230d122e None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:14.651643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb713886-f2f0-4cda-a759-d8c0230d122e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:14.651643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb713886-f2f0-4cda-a759-d8c0230d122e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:14.747309 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-cb713886-f2f0-4cda-a759-d8c0230d122e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:14.749555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-cb713886-f2f0-4cda-a759-d8c0230d122e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:14.807318 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:14.901442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 676/1362] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:22:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:22:15.367185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe40229-9ba5-457e-8928-0698ede2c388 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:22:15.370318 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-cbe40229-9ba5-457e-8928-0698ede2c388 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:15.670783 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe40229-9ba5-457e-8928-0698ede2c388 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:22:15.682928 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:15.868856 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fc4c3a-a55b-4860-a148-e8d495603145 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:22:15.869892 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fc4c3a-a55b-4860-a148-e8d495603145 service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:15.870075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fc4c3a-a55b-4860-a148-e8d495603145 service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:15.871126 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fc4c3a-a55b-4860-a148-e8d495603145 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:15.873286 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-06fc4c3a-a55b-4860-a148-e8d495603145 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:16.034736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fc4c3a-a55b-4860-a148-e8d495603145 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:22:16.038432 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fc4c3a-a55b-4860-a148-e8d495603145 service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:22:16.075365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 687/1363] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:22:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 709 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:22:19.039262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0d66e0-b615-4b9f-bb47-2df71fd391fc None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:19.039975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0d66e0-b615-4b9f-bb47-2df71fd391fc None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:19.039975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0d66e0-b615-4b9f-bb47-2df71fd391fc None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:19.042887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 677/1364] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:22:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:22:19.047544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83cfaa1-4b16-4ef7-a1b5-dec6e25361aa None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:19.048114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83cfaa1-4b16-4ef7-a1b5-dec6e25361aa None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:19.048114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83cfaa1-4b16-4ef7-a1b5-dec6e25361aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:19.070554 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:19.175241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-d83cfaa1-4b16-4ef7-a1b5-dec6e25361aa None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:19.177526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d83cfaa1-4b16-4ef7-a1b5-dec6e25361aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:19.384552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 688/1365] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:22:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 335 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:22:19.923487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591a8c3f-5af2-4c71-8b2e-3631c896c2dd None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:19.924971 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591a8c3f-5af2-4c71-8b2e-3631c896c2dd None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:19.925952 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591a8c3f-5af2-4c71-8b2e-3631c896c2dd None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:19.929011 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 678/1366] 10.222.0.64 () {54 vars in 961 bytes} [Tue Oct 4 14:22:19 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:22:19.937191 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d59286a-1d59-49bd-bc96-6191de7c3672 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:19.937631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d59286a-1d59-49bd-bc96-6191de7c3672 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:19.938108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d59286a-1d59-49bd-bc96-6191de7c3672 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:20.087514 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-9d59286a-1d59-49bd-bc96-6191de7c3672 None None] MFA Rules not processed for user `03081d7d581745c88c12e9d089fcb2d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:20.089230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9d59286a-1d59-49bd-bc96-6191de7c3672 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:20.139437 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:20.273586 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cdb040-dadb-46f3-a4ec-104384c424d9 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:20.274480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cdb040-dadb-46f3-a4ec-104384c424d9 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:20.274480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cdb040-dadb-46f3-a4ec-104384c424d9 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:20.276728 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 679/1367] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:22:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:22:20.285445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 689/1368] 10.222.0.64 () {58 vars in 1058 bytes} [Tue Oct 4 14:22:19 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 349 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:22:20.289095 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6496d3-74a7-4a00-b4cf-d50b796c6cb2 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:20.289388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6496d3-74a7-4a00-b4cf-d50b796c6cb2 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:20.289671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6496d3-74a7-4a00-b4cf-d50b796c6cb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:20.298737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d59286a-1d59-49bd-bc96-6191de7c3672 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:22:20.310845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d59286a-1d59-49bd-bc96-6191de7c3672 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:22:20.373794 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-9f6496d3-74a7-4a00-b4cf-d50b796c6cb2 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:20.375136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9f6496d3-74a7-4a00-b4cf-d50b796c6cb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:20.480239 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5051c86d-0b26-4979-a2a6-830adb6c3edf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:22:20.481117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5051c86d-0b26-4979-a2a6-830adb6c3edf service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:20.481570 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5051c86d-0b26-4979-a2a6-830adb6c3edf service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:20.482036 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5051c86d-0b26-4979-a2a6-830adb6c3edf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:20.484848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5051c86d-0b26-4979-a2a6-830adb6c3edf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:20.522651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:20.547776 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 680/1369] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:22:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:22:20.651240 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5051c86d-0b26-4979-a2a6-830adb6c3edf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:22:20.654828 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5051c86d-0b26-4979-a2a6-830adb6c3edf service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:22:20.696300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 690/1370] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:22:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:22:20.859314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e13bdf7-a682-4f65-aabf-41a5b37bda52 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:20.859826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e13bdf7-a682-4f65-aabf-41a5b37bda52 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:20.860372 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e13bdf7-a682-4f65-aabf-41a5b37bda52 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:20.865239 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 681/1371] 10.222.0.64 () {54 vars in 963 bytes} [Tue Oct 4 14:22:20 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:22:20.872343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83c535d-f65b-4076-bd64-fada804a5f75 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:20.873205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83c535d-f65b-4076-bd64-fada804a5f75 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:20.873973 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83c535d-f65b-4076-bd64-fada804a5f75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:21.053004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-d83c535d-f65b-4076-bd64-fada804a5f75 None None] MFA Rules not processed for user `058c027ade94498fbab7b9347e87c9c3`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:21.055162 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d83c535d-f65b-4076-bd64-fada804a5f75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:21.213224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 691/1372] 10.222.0.64 () {58 vars in 1060 bytes} [Tue Oct 4 14:22:20 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 343 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:22:21.223617 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e13bdf7-a682-4f65-aabf-41a5b37bda52 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:22:21.227131 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7e13bdf7-a682-4f65-aabf-41a5b37bda52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:21.266421 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:21.426882 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e13bdf7-a682-4f65-aabf-41a5b37bda52 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:22:21.543415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6b844f-39e3-480e-adc2-c9320e4131db service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:22:21.544528 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6b844f-39e3-480e-adc2-c9320e4131db service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:21.545034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6b844f-39e3-480e-adc2-c9320e4131db service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:21.545545 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6b844f-39e3-480e-adc2-c9320e4131db service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:21.548082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-df6b844f-39e3-480e-adc2-c9320e4131db service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:21.657038 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:21.717758 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6b844f-39e3-480e-adc2-c9320e4131db service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:22:21.722006 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6b844f-39e3-480e-adc2-c9320e4131db service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:22:21.764833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 682/1373] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:22:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 542 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:22:23.536738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50be829c-7cdf-43d5-b587-5111e4275e71 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:23.537583 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50be829c-7cdf-43d5-b587-5111e4275e71 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:23.537583 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50be829c-7cdf-43d5-b587-5111e4275e71 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:23.539265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 692/1374] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:22:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:22:23.546265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f0e348-c738-4060-b56e-1a15cd49973f None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:23.547003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f0e348-c738-4060-b56e-1a15cd49973f None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:23.547003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f0e348-c738-4060-b56e-1a15cd49973f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:23.639518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-71f0e348-c738-4060-b56e-1a15cd49973f None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:23.641684 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-71f0e348-c738-4060-b56e-1a15cd49973f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:23.819515 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 683/1375] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:22:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:22:24.135351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50be829c-7cdf-43d5-b587-5111e4275e71 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:22:24.138118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-50be829c-7cdf-43d5-b587-5111e4275e71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:24.151508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:24.385565 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50be829c-7cdf-43d5-b587-5111e4275e71 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:22:24.529026 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:24.589118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc159ae-d914-44f3-84dc-9dc2028d5da3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:22:24.590146 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc159ae-d914-44f3-84dc-9dc2028d5da3 service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:24.590634 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc159ae-d914-44f3-84dc-9dc2028d5da3 service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:24.591490 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc159ae-d914-44f3-84dc-9dc2028d5da3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:24.594342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9cc159ae-d914-44f3-84dc-9dc2028d5da3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:24.756578 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc159ae-d914-44f3-84dc-9dc2028d5da3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:22:24.758993 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc159ae-d914-44f3-84dc-9dc2028d5da3 service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:22:24.794944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 693/1376] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:22:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 660 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:22:27.804930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59ec56b-8e15-4aa3-a625-4f0e4c5b18d4 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:27.805843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59ec56b-8e15-4aa3-a625-4f0e4c5b18d4 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:27.805843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59ec56b-8e15-4aa3-a625-4f0e4c5b18d4 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:27.808383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 684/1377] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:22:27 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:22:27.818784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7911a31-3fbf-481e-a61b-5e3f1e8dc214 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:27.819662 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7911a31-3fbf-481e-a61b-5e3f1e8dc214 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:27.819826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7911a31-3fbf-481e-a61b-5e3f1e8dc214 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:27.951337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-b7911a31-3fbf-481e-a61b-5e3f1e8dc214 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:27.952619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b7911a31-3fbf-481e-a61b-5e3f1e8dc214 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:28.013893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:28.088527 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 694/1378] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:22:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:22:28.480480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59ec56b-8e15-4aa3-a625-4f0e4c5b18d4 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:22:28.483268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-b59ec56b-8e15-4aa3-a625-4f0e4c5b18d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:28.548188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:28.855867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59ec56b-8e15-4aa3-a625-4f0e4c5b18d4 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:22:29.102066 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:29.124487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee36ce7f-c2ff-4643-ba62-cc8ab1d2079b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:22:29.125818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee36ce7f-c2ff-4643-ba62-cc8ab1d2079b service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:29.126539 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee36ce7f-c2ff-4643-ba62-cc8ab1d2079b service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:29.127208 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee36ce7f-c2ff-4643-ba62-cc8ab1d2079b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:29.130752 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ee36ce7f-c2ff-4643-ba62-cc8ab1d2079b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:29.315904 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee36ce7f-c2ff-4643-ba62-cc8ab1d2079b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:22:29.317831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee36ce7f-c2ff-4643-ba62-cc8ab1d2079b service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:22:29.358135 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 685/1379] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:22:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 878 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:22:32.903913 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17593ec7-9a0e-41ba-b952-daa9c92faf1f None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:32.905091 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17593ec7-9a0e-41ba-b952-daa9c92faf1f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:32.905545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17593ec7-9a0e-41ba-b952-daa9c92faf1f None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:32.910010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 695/1380] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:22:32 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:22:32.916681 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c40279-4491-474a-b785-d1782b2d40f2 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:32.917379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c40279-4491-474a-b785-d1782b2d40f2 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:32.917584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c40279-4491-474a-b785-d1782b2d40f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:33.022442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-54c40279-4491-474a-b785-d1782b2d40f2 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:33.023738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-54c40279-4491-474a-b785-d1782b2d40f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:33.107250 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:33.158092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 686/1381] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:22:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:22:33.726587 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17593ec7-9a0e-41ba-b952-daa9c92faf1f None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:22:33.728239 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-17593ec7-9a0e-41ba-b952-daa9c92faf1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:34.027679 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17593ec7-9a0e-41ba-b952-daa9c92faf1f None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:22:34.063766 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:34.250364 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc31238-dcb1-427b-8920-f66c003909dc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:22:34.251201 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc31238-dcb1-427b-8920-f66c003909dc service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:34.251564 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc31238-dcb1-427b-8920-f66c003909dc service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:34.251947 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc31238-dcb1-427b-8920-f66c003909dc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:34.254370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7cc31238-dcb1-427b-8920-f66c003909dc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:34.402542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc31238-dcb1-427b-8920-f66c003909dc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:22:34.405136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc31238-dcb1-427b-8920-f66c003909dc service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:22:34.436076 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 696/1382] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:22:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 710 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:22:36.900305 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f281cdb2-9df4-4fa5-8ef0-d1cd154864e7 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:36.901192 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f281cdb2-9df4-4fa5-8ef0-d1cd154864e7 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:36.901192 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f281cdb2-9df4-4fa5-8ef0-d1cd154864e7 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:36.902539 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 687/1383] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:22:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:22:36.910938 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff4d9a-93bc-4114-89a0-f6566e395b7d None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:36.911501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff4d9a-93bc-4114-89a0-f6566e395b7d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:36.911881 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff4d9a-93bc-4114-89a0-f6566e395b7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:36.982678 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:37.015809 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-5fff4d9a-93bc-4114-89a0-f6566e395b7d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:37.017225 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5fff4d9a-93bc-4114-89a0-f6566e395b7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:37.195020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 697/1384] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:22:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:22:37.505461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f281cdb2-9df4-4fa5-8ef0-d1cd154864e7 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:22:37.508247 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-f281cdb2-9df4-4fa5-8ef0-d1cd154864e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:37.763264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f281cdb2-9df4-4fa5-8ef0-d1cd154864e7 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:22:37.852643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:37.969759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c96466-445a-482b-a548-61488674ba52 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:22:37.970888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c96466-445a-482b-a548-61488674ba52 service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:37.971355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c96466-445a-482b-a548-61488674ba52 service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:37.971869 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c96466-445a-482b-a548-61488674ba52 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:37.975165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-82c96466-445a-482b-a548-61488674ba52 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:38.114073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c96466-445a-482b-a548-61488674ba52 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:22:38.115944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c96466-445a-482b-a548-61488674ba52 service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:22:38.139556 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 688/1385] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:22:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 635 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:22:39.681387 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6408b8-c258-41e7-acd3-6ce0bc8eea48 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:39.682180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6408b8-c258-41e7-acd3-6ce0bc8eea48 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:39.682439 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6408b8-c258-41e7-acd3-6ce0bc8eea48 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:39.685462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 698/1386] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:22:39 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:22:39.692209 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761b1286-50dc-4aed-8e55-717969459e92 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:39.693088 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761b1286-50dc-4aed-8e55-717969459e92 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:39.693088 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761b1286-50dc-4aed-8e55-717969459e92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:39.779459 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-761b1286-50dc-4aed-8e55-717969459e92 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:39.780942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-761b1286-50dc-4aed-8e55-717969459e92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:39.784574 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:39.913129 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 689/1387] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:22:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:22:40.231370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e49a64-86fb-4c2a-9285-96f0f89561a8 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:40.231680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e49a64-86fb-4c2a-9285-96f0f89561a8 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:40.231953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e49a64-86fb-4c2a-9285-96f0f89561a8 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:40.233432 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 699/1388] 10.222.0.64 () {54 vars in 962 bytes} [Tue Oct 4 14:22:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:22:40.241347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba84e54-06fe-43bf-bc0b-85130db392fb None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:40.241830 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba84e54-06fe-43bf-bc0b-85130db392fb None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:40.242283 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba84e54-06fe-43bf-bc0b-85130db392fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:40.341550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-0ba84e54-06fe-43bf-bc0b-85130db392fb None None] MFA Rules not processed for user `35a5a4ef40204e158649ca25583bbde7`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:40.342799 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0ba84e54-06fe-43bf-bc0b-85130db392fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:40.446136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:40.461445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 690/1389] 10.222.0.64 () {58 vars in 1059 bytes} [Tue Oct 4 14:22:40 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:22:40.462835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e49a64-86fb-4c2a-9285-96f0f89561a8 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:22:40.464664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-e7e49a64-86fb-4c2a-9285-96f0f89561a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:40.597425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e49a64-86fb-4c2a-9285-96f0f89561a8 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:22:40.598593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f626c446-ba24-42f7-b9b3-c2de1171bab4 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:40.598939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f626c446-ba24-42f7-b9b3-c2de1171bab4 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:40.599308 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f626c446-ba24-42f7-b9b3-c2de1171bab4 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:40.601314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 691/1390] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:22:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:22:40.607435 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7574caf9-4012-4ce9-af92-62d2f2a455bd None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:40.607714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7574caf9-4012-4ce9-af92-62d2f2a455bd None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:40.607988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7574caf9-4012-4ce9-af92-62d2f2a455bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:40.615322 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:40.699022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9d9b11-46b5-4124-add5-e7d835d73984 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35a5a4ef40204e158649ca25583bbde7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:22:40.700003 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9d9b11-46b5-4124-add5-e7d835d73984 service glance] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:40.700003 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9d9b11-46b5-4124-add5-e7d835d73984 service glance] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:40.700352 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9d9b11-46b5-4124-add5-e7d835d73984 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:40.701779 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-0d9d9b11-46b5-4124-add5-e7d835d73984 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:40.713690 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-7574caf9-4012-4ce9-af92-62d2f2a455bd None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:40.715826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7574caf9-4012-4ce9-af92-62d2f2a455bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:40.860049 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9d9b11-46b5-4124-add5-e7d835d73984 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:22:40.861798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9d9b11-46b5-4124-add5-e7d835d73984 service glance] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:22:40.880893 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 692/1391] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:22:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:22:40.885510 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 700/1392] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:22:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:22:41.207902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7574caf9-4012-4ce9-af92-62d2f2a455bd None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:22:41.209324 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7574caf9-4012-4ce9-af92-62d2f2a455bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:41.338441 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:41.403083 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7574caf9-4012-4ce9-af92-62d2f2a455bd None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:22:41.610536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae4d5ea-ef3d-4502-b824-a8c1bafc56ce service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:22:41.612656 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae4d5ea-ef3d-4502-b824-a8c1bafc56ce service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:41.613333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae4d5ea-ef3d-4502-b824-a8c1bafc56ce service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:41.613748 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae4d5ea-ef3d-4502-b824-a8c1bafc56ce service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:41.616670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7ae4d5ea-ef3d-4502-b824-a8c1bafc56ce service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:41.736857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:41.769000 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae4d5ea-ef3d-4502-b824-a8c1bafc56ce service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:22:41.771608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae4d5ea-ef3d-4502-b824-a8c1bafc56ce service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:22:41.800566 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5761e391-9f75-4c26-a924-d32bfaea5415 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:41.801240 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5761e391-9f75-4c26-a924-d32bfaea5415 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:41.801492 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5761e391-9f75-4c26-a924-d32bfaea5415 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:41.802672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 693/1393] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:22:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 595 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:22:41.870586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:41.898956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-5761e391-9f75-4c26-a924-d32bfaea5415 None None] MFA Rules not processed for user `34afa4aac10c4a018ca05a0ba34c940a`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:41.900221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5761e391-9f75-4c26-a924-d32bfaea5415 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:42.039295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 701/1394] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Oct 4 14:22:41 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:22:42.104815 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925bcd91-bfa1-47d2-b66e-877d01b0eab8 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:42.105190 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925bcd91-bfa1-47d2-b66e-877d01b0eab8 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:42.105535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925bcd91-bfa1-47d2-b66e-877d01b0eab8 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:42.107284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 694/1395] 10.222.0.64 () {54 vars in 943 bytes} [Tue Oct 4 14:22:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:22:42.116067 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d79932-21f3-4a86-be4c-7e040010e340 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:42.116416 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d79932-21f3-4a86-be4c-7e040010e340 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:42.116639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d79932-21f3-4a86-be4c-7e040010e340 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:42.207825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-13d79932-21f3-4a86-be4c-7e040010e340 None None] MFA Rules not processed for user `5980e353164c4873bcdd2af847e0a1f1`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:42.208999 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-13d79932-21f3-4a86-be4c-7e040010e340 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:42.289936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:42.343267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 702/1396] 10.222.0.64 () {58 vars in 1040 bytes} [Tue Oct 4 14:22:42 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:22:42.354311 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925bcd91-bfa1-47d2-b66e-877d01b0eab8 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:22:42.355972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-925bcd91-bfa1-47d2-b66e-877d01b0eab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:42.558167 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925bcd91-bfa1-47d2-b66e-877d01b0eab8 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:22:42.636539 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:42.701576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9719b030-ed2f-4003-93f9-66832f793340 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5980e353164c4873bcdd2af847e0a1f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:22:42.702284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9719b030-ed2f-4003-93f9-66832f793340 service swift] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:42.702626 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9719b030-ed2f-4003-93f9-66832f793340 service swift] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:42.702990 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9719b030-ed2f-4003-93f9-66832f793340 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:42.705660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9719b030-ed2f-4003-93f9-66832f793340 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:42.830335 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9719b030-ed2f-4003-93f9-66832f793340 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:22:42.832177 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9719b030-ed2f-4003-93f9-66832f793340 service swift] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:22:42.842955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 695/1397] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Oct 4 14:22:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 489 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 14:22:44.757945 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8c9049-8958-496e-9f2b-f9d1d609dc77 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:44.758903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8c9049-8958-496e-9f2b-f9d1d609dc77 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:44.758903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8c9049-8958-496e-9f2b-f9d1d609dc77 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:44.759848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 703/1398] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:22:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:22:44.770483 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9bbf13-a1fd-4bf4-8f60-2ddf13944806 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:44.771385 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9bbf13-a1fd-4bf4-8f60-2ddf13944806 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:44.771534 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9bbf13-a1fd-4bf4-8f60-2ddf13944806 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:44.904476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-8e9bbf13-a1fd-4bf4-8f60-2ddf13944806 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:44.906925 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8e9bbf13-a1fd-4bf4-8f60-2ddf13944806 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:45.006454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:45.128007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 696/1399] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:22:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 360 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:22:45.679195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8c9049-8958-496e-9f2b-f9d1d609dc77 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:22:45.682153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-da8c9049-8958-496e-9f2b-f9d1d609dc77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:45.950220 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8c9049-8958-496e-9f2b-f9d1d609dc77 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:22:46.025910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:46.131943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3340d405-1120-4d00-81e2-54f3c42322fa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:22:46.132589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3340d405-1120-4d00-81e2-54f3c42322fa service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:46.132864 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3340d405-1120-4d00-81e2-54f3c42322fa service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:46.133129 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3340d405-1120-4d00-81e2-54f3c42322fa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:46.134814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-3340d405-1120-4d00-81e2-54f3c42322fa service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:46.307189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3340d405-1120-4d00-81e2-54f3c42322fa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:22:46.310913 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3340d405-1120-4d00-81e2-54f3c42322fa service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:22:46.353016 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 704/1400] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:22:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 675 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:22:48.830922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18f7d51-9991-459f-9320-0454e950c92a None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:48.831532 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18f7d51-9991-459f-9320-0454e950c92a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:48.831532 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18f7d51-9991-459f-9320-0454e950c92a None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:48.832821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 697/1401] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:22:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:22:48.838920 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a18b39-e80b-47f7-b9ab-45cbe9e0c6ce None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:48.839502 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a18b39-e80b-47f7-b9ab-45cbe9e0c6ce None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:48.839713 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a18b39-e80b-47f7-b9ab-45cbe9e0c6ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:48.944467 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:48.951434 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-27a18b39-e80b-47f7-b9ab-45cbe9e0c6ce None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:48.958127 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-27a18b39-e80b-47f7-b9ab-45cbe9e0c6ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:49.223601 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 705/1402] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:22:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 386 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:22:49.588585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18f7d51-9991-459f-9320-0454e950c92a None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:22:49.593941 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-b18f7d51-9991-459f-9320-0454e950c92a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:49.903132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18f7d51-9991-459f-9320-0454e950c92a None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:22:49.965106 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:50.130507 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e77c35-49bf-4722-8600-c12026de19d3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:22:50.131761 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e77c35-49bf-4722-8600-c12026de19d3 service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:50.132432 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e77c35-49bf-4722-8600-c12026de19d3 service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:50.132919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e77c35-49bf-4722-8600-c12026de19d3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:50.136343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-51e77c35-49bf-4722-8600-c12026de19d3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:50.300364 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e77c35-49bf-4722-8600-c12026de19d3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:22:50.302041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e77c35-49bf-4722-8600-c12026de19d3 service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:22:50.329339 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 698/1403] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:22:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 742 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:22:53.249798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ace8d55-e9d8-4f96-b254-57d61d7c446b None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:53.250737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ace8d55-e9d8-4f96-b254-57d61d7c446b None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:53.250737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ace8d55-e9d8-4f96-b254-57d61d7c446b None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:53.256435 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 706/1404] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:22:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:22:53.258011 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ba4671-7839-47a1-9b6e-ee44cbfd1ef9 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:53.258758 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ba4671-7839-47a1-9b6e-ee44cbfd1ef9 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:53.258758 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ba4671-7839-47a1-9b6e-ee44cbfd1ef9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:53.340570 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:53.387605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-68ba4671-7839-47a1-9b6e-ee44cbfd1ef9 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:53.389529 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-68ba4671-7839-47a1-9b6e-ee44cbfd1ef9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:53.592491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 699/1405] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:22:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 335 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:22:53.929093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ace8d55-e9d8-4f96-b254-57d61d7c446b None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:22:53.930346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-4ace8d55-e9d8-4f96-b254-57d61d7c446b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:54.143303 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:54.165425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ace8d55-e9d8-4f96-b254-57d61d7c446b None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:22:54.393305 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2e8b08-ec71-4cb1-a795-45c70f6f928b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:22:54.394038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2e8b08-ec71-4cb1-a795-45c70f6f928b service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:54.394327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2e8b08-ec71-4cb1-a795-45c70f6f928b service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:54.394582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2e8b08-ec71-4cb1-a795-45c70f6f928b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:54.396276 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6a2e8b08-ec71-4cb1-a795-45c70f6f928b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:54.528950 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2e8b08-ec71-4cb1-a795-45c70f6f928b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:22:54.530962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2e8b08-ec71-4cb1-a795-45c70f6f928b service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:22:54.536794 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:54.569135 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 707/1406] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:22:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 641 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:22:57.144167 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d510717-29eb-4c44-a2e8-827566ffbaf6 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:57.145125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d510717-29eb-4c44-a2e8-827566ffbaf6 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:57.145343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d510717-29eb-4c44-a2e8-827566ffbaf6 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:57.147021 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 700/1407] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:22:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:22:57.152658 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe127cf-5beb-4970-ac18-95ce5720aed5 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:57.153229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe127cf-5beb-4970-ac18-95ce5720aed5 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:57.153229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe127cf-5beb-4970-ac18-95ce5720aed5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:57.264943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-dfe127cf-5beb-4970-ac18-95ce5720aed5 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:22:57.266807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-dfe127cf-5beb-4970-ac18-95ce5720aed5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:57.448158 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 708/1408] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:22:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 295 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:22:57.887251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d510717-29eb-4c44-a2e8-827566ffbaf6 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:22:57.889041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-1d510717-29eb-4c44-a2e8-827566ffbaf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:58.063387 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:58.129424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d510717-29eb-4c44-a2e8-827566ffbaf6 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:22:58.348992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487a137e-6292-413d-8318-589886567fa2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:22:58.351107 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487a137e-6292-413d-8318-589886567fa2 service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:22:58.351852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487a137e-6292-413d-8318-589886567fa2 service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:22:58.352574 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487a137e-6292-413d-8318-589886567fa2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:22:58.355712 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-487a137e-6292-413d-8318-589886567fa2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:22:58.489388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:22:58.518344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487a137e-6292-413d-8318-589886567fa2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:22:58.520129 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487a137e-6292-413d-8318-589886567fa2 service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:22:58.541367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 701/1409] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:22:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 655 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:23:07.441584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=70 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:08.538955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=12 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:09.012052 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35767ab-8892-4cb6-912a-1683e08a7cda None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:09.012700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35767ab-8892-4cb6-912a-1683e08a7cda None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:09.012700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35767ab-8892-4cb6-912a-1683e08a7cda None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:09.014024 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 709/1410] 10.222.0.64 () {54 vars in 918 bytes} [Tue Oct 4 14:23:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:09.020616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e60a786-8db0-4ad1-a6f3-02cac82e7697 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:09.020906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e60a786-8db0-4ad1-a6f3-02cac82e7697 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:09.021213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e60a786-8db0-4ad1-a6f3-02cac82e7697 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:09.123527 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-2e60a786-8db0-4ad1-a6f3-02cac82e7697 None None] MFA Rules not processed for user `436deb3d60c54a799700986519b2ef68`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:09.124961 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-2e60a786-8db0-4ad1-a6f3-02cac82e7697 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:09.264946 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 702/1411] 10.222.0.64 () {58 vars in 1015 bytes} [Tue Oct 4 14:23:09 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:23:09.281597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a4bce6-17c7-4854-8117-1756d3638614 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:09.281893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a4bce6-17c7-4854-8117-1756d3638614 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:09.282133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a4bce6-17c7-4854-8117-1756d3638614 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:09.284658 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 710/1412] 10.222.0.64 () {54 vars in 918 bytes} [Tue Oct 4 14:23:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:09.290560 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ac21ef-4ba2-43ac-b4fd-298b8798cd44 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:09.290833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ac21ef-4ba2-43ac-b4fd-298b8798cd44 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:09.291151 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ac21ef-4ba2-43ac-b4fd-298b8798cd44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:09.386569 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:09.390925 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-c9ac21ef-4ba2-43ac-b4fd-298b8798cd44 None None] MFA Rules not processed for user `03081d7d581745c88c12e9d089fcb2d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:09.392763 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c9ac21ef-4ba2-43ac-b4fd-298b8798cd44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:09.531426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 703/1413] 10.222.0.64 () {58 vars in 1015 bytes} [Tue Oct 4 14:23:09 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:23:12.394031 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bc23ca-f1b5-401a-ac71-35a0aa6827b4 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:12.394675 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bc23ca-f1b5-401a-ac71-35a0aa6827b4 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:12.394810 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bc23ca-f1b5-401a-ac71-35a0aa6827b4 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:12.396186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 711/1414] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 14:23:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:12.402361 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e5b8d8-90d9-4578-bb30-03e56576f1e6 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:12.403204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e5b8d8-90d9-4578-bb30-03e56576f1e6 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:12.403204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e5b8d8-90d9-4578-bb30-03e56576f1e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:12.483901 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-34e5b8d8-90d9-4578-bb30-03e56576f1e6 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:12.485194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-34e5b8d8-90d9-4578-bb30-03e56576f1e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:12.551834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:12.602525 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 704/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 14:23:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:23:12.906300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67bc23ca-f1b5-401a-ac71-35a0aa6827b4 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:12.908832 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-67bc23ca-f1b5-401a-ac71-35a0aa6827b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:13.089824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67bc23ca-f1b5-401a-ac71-35a0aa6827b4 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:13.186606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:13.246603 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef20f08-23bf-4dae-847a-c5a989ec3d8d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:13.247661 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef20f08-23bf-4dae-847a-c5a989ec3d8d service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:13.248108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef20f08-23bf-4dae-847a-c5a989ec3d8d service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:13.248610 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef20f08-23bf-4dae-847a-c5a989ec3d8d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:13.252252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-3ef20f08-23bf-4dae-847a-c5a989ec3d8d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:13.421723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef20f08-23bf-4dae-847a-c5a989ec3d8d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:13.424182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef20f08-23bf-4dae-847a-c5a989ec3d8d service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:13.459710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 712/1416] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 554 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:23:14.735501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8184e0-7a6f-4b84-bff4-a7b14bbda5c8 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:14.736119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8184e0-7a6f-4b84-bff4-a7b14bbda5c8 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:14.736119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8184e0-7a6f-4b84-bff4-a7b14bbda5c8 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:14.738660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 705/1417] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 14:23:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:14.745877 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c928f2-0513-47ad-925a-b9c71c49b4cb None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:14.746474 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c928f2-0513-47ad-925a-b9c71c49b4cb None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:14.746474 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c928f2-0513-47ad-925a-b9c71c49b4cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:14.837790 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-54c928f2-0513-47ad-925a-b9c71c49b4cb None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:14.839148 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-54c928f2-0513-47ad-925a-b9c71c49b4cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:14.870819 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:14.954640 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 713/1418] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 14:23:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:23:15.149394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8184e0-7a6f-4b84-bff4-a7b14bbda5c8 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:15.151005 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7d8184e0-7a6f-4b84-bff4-a7b14bbda5c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:15.379061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8184e0-7a6f-4b84-bff4-a7b14bbda5c8 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:15.450491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:15.539143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c68803-bede-493b-bf40-452fc67dbd55 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:15.539764 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c68803-bede-493b-bf40-452fc67dbd55 service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:15.540045 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c68803-bede-493b-bf40-452fc67dbd55 service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:15.540351 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c68803-bede-493b-bf40-452fc67dbd55 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:15.542080 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-76c68803-bede-493b-bf40-452fc67dbd55 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:15.646659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c68803-bede-493b-bf40-452fc67dbd55 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:15.648302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c68803-bede-493b-bf40-452fc67dbd55 service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:15.670408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 706/1419] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 521 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:23:16.929331 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb4320-af93-46cc-9179-d623cb85f244 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:16.930203 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb4320-af93-46cc-9179-d623cb85f244 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:16.930203 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb4320-af93-46cc-9179-d623cb85f244 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:16.931255 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 714/1420] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 14:23:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:16.938312 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f594f71-555e-4adb-bcba-aec027da8f73 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:16.938888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f594f71-555e-4adb-bcba-aec027da8f73 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:16.938888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f594f71-555e-4adb-bcba-aec027da8f73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:17.002352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:17.031321 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-5f594f71-555e-4adb-bcba-aec027da8f73 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:17.032597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-5f594f71-555e-4adb-bcba-aec027da8f73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:17.156503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 707/1421] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 14:23:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:23:17.313276 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eb4320-af93-46cc-9179-d623cb85f244 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:17.316427 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-74eb4320-af93-46cc-9179-d623cb85f244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:17.504787 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eb4320-af93-46cc-9179-d623cb85f244 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:17.520925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:17.677625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e33981-e713-4118-ad33-b6b842b1a489 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:17.678456 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e33981-e713-4118-ad33-b6b842b1a489 service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:17.678802 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e33981-e713-4118-ad33-b6b842b1a489 service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:17.679229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e33981-e713-4118-ad33-b6b842b1a489 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:17.681627 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b8e33981-e713-4118-ad33-b6b842b1a489 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:17.808504 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e33981-e713-4118-ad33-b6b842b1a489 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:17.810174 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e33981-e713-4118-ad33-b6b842b1a489 service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:17.837661 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 715/1422] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 526 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:23:19.217315 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f3493f-1bd6-4bbd-a4b1-e75a030d4fb0 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:19.218231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f3493f-1bd6-4bbd-a4b1-e75a030d4fb0 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:19.218231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f3493f-1bd6-4bbd-a4b1-e75a030d4fb0 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:19.219841 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 708/1423] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 14:23:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:19.228012 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10293cc0-3708-4c43-8994-f9d4fded6540 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:19.228711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10293cc0-3708-4c43-8994-f9d4fded6540 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:19.228711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10293cc0-3708-4c43-8994-f9d4fded6540 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:19.250781 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:19.321544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-10293cc0-3708-4c43-8994-f9d4fded6540 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:19.322864 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-10293cc0-3708-4c43-8994-f9d4fded6540 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:19.446596 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 716/1424] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 14:23:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:23:19.612762 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f3493f-1bd6-4bbd-a4b1-e75a030d4fb0 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:19.614341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-f6f3493f-1bd6-4bbd-a4b1-e75a030d4fb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:19.883222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:19.928343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f3493f-1bd6-4bbd-a4b1-e75a030d4fb0 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:20.069021 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addd87c8-763d-4d28-9687-8ea1db35c0a3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:20.069705 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd87c8-763d-4d28-9687-8ea1db35c0a3 service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:20.070075 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd87c8-763d-4d28-9687-8ea1db35c0a3 service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:20.070407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd87c8-763d-4d28-9687-8ea1db35c0a3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:20.072298 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-addd87c8-763d-4d28-9687-8ea1db35c0a3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:20.178733 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:20.197707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addd87c8-763d-4d28-9687-8ea1db35c0a3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:20.199341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addd87c8-763d-4d28-9687-8ea1db35c0a3 service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:20.221632 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 709/1425] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 610 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:23:21.452396 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1ff313-5ac5-46ee-8673-4b22366e2896 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:21.453174 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1ff313-5ac5-46ee-8673-4b22366e2896 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:21.453174 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1ff313-5ac5-46ee-8673-4b22366e2896 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:21.454427 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 717/1426] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 14:23:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:21.462419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51941111-9f4d-4b05-80d7-798fa9909167 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:21.463162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51941111-9f4d-4b05-80d7-798fa9909167 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:21.463162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51941111-9f4d-4b05-80d7-798fa9909167 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:21.554637 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-51941111-9f4d-4b05-80d7-798fa9909167 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:21.555903 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-51941111-9f4d-4b05-80d7-798fa9909167 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:21.667552 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 710/1427] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 14:23:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:23:21.829287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1ff313-5ac5-46ee-8673-4b22366e2896 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:21.831085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5f1ff313-5ac5-46ee-8673-4b22366e2896 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:21.938507 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:22.028201 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1ff313-5ac5-46ee-8673-4b22366e2896 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:22.248539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bac0ae-1dc1-4f3e-97c3-1e09c1604c81 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:22.249426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bac0ae-1dc1-4f3e-97c3-1e09c1604c81 service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:22.249906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bac0ae-1dc1-4f3e-97c3-1e09c1604c81 service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:22.250382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bac0ae-1dc1-4f3e-97c3-1e09c1604c81 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:22.253710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-23bac0ae-1dc1-4f3e-97c3-1e09c1604c81 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:22.354412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:22.435912 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23bac0ae-1dc1-4f3e-97c3-1e09c1604c81 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:22.439079 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23bac0ae-1dc1-4f3e-97c3-1e09c1604c81 service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:22.478089 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 718/1428] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 649 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:23:23.722166 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ea90a2-8471-4712-b7bf-25239a6f0951 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:23.722749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ea90a2-8471-4712-b7bf-25239a6f0951 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:23.722749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ea90a2-8471-4712-b7bf-25239a6f0951 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:23.723908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 711/1429] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 14:23:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:23.731146 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab79ebe-b791-4b06-abf6-a74ca8c5af7b None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:23.731758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab79ebe-b791-4b06-abf6-a74ca8c5af7b None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:23.731758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab79ebe-b791-4b06-abf6-a74ca8c5af7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:23.828266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-fab79ebe-b791-4b06-abf6-a74ca8c5af7b None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:23.829741 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-fab79ebe-b791-4b06-abf6-a74ca8c5af7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:23.966422 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 719/1430] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 14:23:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:23:24.133439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ea90a2-8471-4712-b7bf-25239a6f0951 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:24.135166 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-13ea90a2-8471-4712-b7bf-25239a6f0951 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:24.194771 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:24.374008 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ea90a2-8471-4712-b7bf-25239a6f0951 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:24.511640 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a888e8ef-0b39-4bc9-b7a4-408650660145 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:24.512329 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a888e8ef-0b39-4bc9-b7a4-408650660145 service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:24.512688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a888e8ef-0b39-4bc9-b7a4-408650660145 service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:24.513002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a888e8ef-0b39-4bc9-b7a4-408650660145 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:24.514736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a888e8ef-0b39-4bc9-b7a4-408650660145 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:24.524564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:24.660966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a888e8ef-0b39-4bc9-b7a4-408650660145 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:24.662633 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a888e8ef-0b39-4bc9-b7a4-408650660145 service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:24.684401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 712/1431] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 551 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:23:25.840240 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681943a8-4fbb-4590-a9e7-f2875edeca1c None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:25.840904 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681943a8-4fbb-4590-a9e7-f2875edeca1c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:25.840904 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681943a8-4fbb-4590-a9e7-f2875edeca1c None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:25.842176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 720/1432] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 14:23:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:25.852419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f319c3-c35b-4d53-afb9-4e161923014a None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:25.853280 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f319c3-c35b-4d53-afb9-4e161923014a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:25.853477 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f319c3-c35b-4d53-afb9-4e161923014a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:25.949868 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-78f319c3-c35b-4d53-afb9-4e161923014a None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:25.951177 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-78f319c3-c35b-4d53-afb9-4e161923014a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:26.037839 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:26.073881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 713/1433] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 14:23:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:23:26.233784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681943a8-4fbb-4590-a9e7-f2875edeca1c None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:26.235520 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-681943a8-4fbb-4590-a9e7-f2875edeca1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:26.256592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:26.415030 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681943a8-4fbb-4590-a9e7-f2875edeca1c None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:26.518261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:26.549696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab3ce53-55e6-49df-bb25-5784f3dbd9d2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:26.550460 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab3ce53-55e6-49df-bb25-5784f3dbd9d2 service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:26.550839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab3ce53-55e6-49df-bb25-5784f3dbd9d2 service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:26.551213 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab3ce53-55e6-49df-bb25-5784f3dbd9d2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:26.553112 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-4ab3ce53-55e6-49df-bb25-5784f3dbd9d2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:26.664547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab3ce53-55e6-49df-bb25-5784f3dbd9d2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:26.666333 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab3ce53-55e6-49df-bb25-5784f3dbd9d2 service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:26.688600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 721/1434] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 456 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:23:27.879248 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ffb666-e4df-489b-83cc-245d832119ec None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:27.879835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ffb666-e4df-489b-83cc-245d832119ec None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:27.879835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ffb666-e4df-489b-83cc-245d832119ec None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:27.881153 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 714/1435] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 14:23:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:27.888224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f182730-6030-44f6-904b-7b2aaf2e6b56 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:27.889105 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f182730-6030-44f6-904b-7b2aaf2e6b56 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:27.889281 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f182730-6030-44f6-904b-7b2aaf2e6b56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:27.985458 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-1f182730-6030-44f6-904b-7b2aaf2e6b56 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:27.986802 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1f182730-6030-44f6-904b-7b2aaf2e6b56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:28.045062 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:28.113313 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 722/1436] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 14:23:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:23:28.275320 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ffb666-e4df-489b-83cc-245d832119ec None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:28.276833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-30ffb666-e4df-489b-83cc-245d832119ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:28.489546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ffb666-e4df-489b-83cc-245d832119ec None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:28.557574 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:28.631368 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df00b416-6f12-45bb-9d6a-fbfe297be959 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:28.632034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df00b416-6f12-45bb-9d6a-fbfe297be959 service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:28.632348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df00b416-6f12-45bb-9d6a-fbfe297be959 service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:28.632663 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df00b416-6f12-45bb-9d6a-fbfe297be959 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:28.634410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-df00b416-6f12-45bb-9d6a-fbfe297be959 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:28.750026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df00b416-6f12-45bb-9d6a-fbfe297be959 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:28.752408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df00b416-6f12-45bb-9d6a-fbfe297be959 service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:28.774582 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 715/1437] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 500 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:23:29.905360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5f2d36-ad29-4cdf-a793-fb04eaa63387 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:29.906473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5f2d36-ad29-4cdf-a793-fb04eaa63387 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:29.906473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5f2d36-ad29-4cdf-a793-fb04eaa63387 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:29.907587 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 723/1438] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 14:23:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:23:29.916062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f473ec34-c73c-4b41-afba-752f229abc0e None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:29.920815 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f473ec34-c73c-4b41-afba-752f229abc0e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:29.920815 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f473ec34-c73c-4b41-afba-752f229abc0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:30.013661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-f473ec34-c73c-4b41-afba-752f229abc0e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:30.014893 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-f473ec34-c73c-4b41-afba-752f229abc0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:30.017552 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:30.125711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 716/1439] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 14:23:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:23:30.287786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5f2d36-ad29-4cdf-a793-fb04eaa63387 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:30.289240 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9b5f2d36-ad29-4cdf-a793-fb04eaa63387 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:30.453426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5f2d36-ad29-4cdf-a793-fb04eaa63387 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:30.488973 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:30.597352 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0b7448-0f78-408b-aff5-5e7dfc808fbb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:30.597874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0b7448-0f78-408b-aff5-5e7dfc808fbb service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:30.598145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0b7448-0f78-408b-aff5-5e7dfc808fbb service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:30.598411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0b7448-0f78-408b-aff5-5e7dfc808fbb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:30.600067 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5d0b7448-0f78-408b-aff5-5e7dfc808fbb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:30.710232 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0b7448-0f78-408b-aff5-5e7dfc808fbb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:30.711839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0b7448-0f78-408b-aff5-5e7dfc808fbb service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:30.730822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 724/1440] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:23:32.885928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b31703b-abb1-4c85-a1e2-22d200db5b11 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:32.886492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b31703b-abb1-4c85-a1e2-22d200db5b11 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:32.886492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b31703b-abb1-4c85-a1e2-22d200db5b11 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:32.887810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 717/1441] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 14:23:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:32.893631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36360621-e734-41b9-90ea-92dbbf039236 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:32.894186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36360621-e734-41b9-90ea-92dbbf039236 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:32.894350 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36360621-e734-41b9-90ea-92dbbf039236 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:32.938594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:32.972504 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-36360621-e734-41b9-90ea-92dbbf039236 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:32.973754 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-36360621-e734-41b9-90ea-92dbbf039236 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:33.074774 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 725/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 14:23:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:23:33.364473 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b31703b-abb1-4c85-a1e2-22d200db5b11 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:33.366040 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9b31703b-abb1-4c85-a1e2-22d200db5b11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:33.537555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b31703b-abb1-4c85-a1e2-22d200db5b11 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:33.572672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:33.632345 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c26826-eba8-4942-a093-42f57ce0feb7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:33.632983 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c26826-eba8-4942-a093-42f57ce0feb7 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:33.633264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c26826-eba8-4942-a093-42f57ce0feb7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:33.633538 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c26826-eba8-4942-a093-42f57ce0feb7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:33.635202 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-15c26826-eba8-4942-a093-42f57ce0feb7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:33.756040 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c26826-eba8-4942-a093-42f57ce0feb7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:33.757673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c26826-eba8-4942-a093-42f57ce0feb7 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:33.785266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 718/1443] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:23:36.209861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b374573-a5e7-44df-b2f1-1ebaf52e14fc None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:36.210504 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b374573-a5e7-44df-b2f1-1ebaf52e14fc None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:36.210504 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b374573-a5e7-44df-b2f1-1ebaf52e14fc None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:36.211788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 726/1444] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 14:23:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:36.217220 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dcc938-30e3-4eca-843c-ca67cef43001 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:36.217677 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dcc938-30e3-4eca-843c-ca67cef43001 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:36.217782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dcc938-30e3-4eca-843c-ca67cef43001 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:36.285240 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-f8dcc938-30e3-4eca-843c-ca67cef43001 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:36.286452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-f8dcc938-30e3-4eca-843c-ca67cef43001 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:36.299713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:36.390105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 719/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 14:23:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:23:36.679148 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b374573-a5e7-44df-b2f1-1ebaf52e14fc None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:36.680715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7b374573-a5e7-44df-b2f1-1ebaf52e14fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:36.790870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b374573-a5e7-44df-b2f1-1ebaf52e14fc None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:36.803890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:36.879923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5c6034-2784-4a31-9917-b20559e8c296 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:36.880569 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5c6034-2784-4a31-9917-b20559e8c296 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:36.880848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5c6034-2784-4a31-9917-b20559e8c296 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:36.881112 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5c6034-2784-4a31-9917-b20559e8c296 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:36.882729 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7d5c6034-2784-4a31-9917-b20559e8c296 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:37.004823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5c6034-2784-4a31-9917-b20559e8c296 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:37.006197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5c6034-2784-4a31-9917-b20559e8c296 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:37.026254 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 727/1446] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:23:39.638188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f881b3b3-4224-470f-8969-aea988e0604a None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:39.638757 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f881b3b3-4224-470f-8969-aea988e0604a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:39.638757 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f881b3b3-4224-470f-8969-aea988e0604a None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:39.640136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 720/1447] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 14:23:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:39.647490 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2adf72-95f6-4b40-85b6-b980d813e0cd None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:39.648024 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2adf72-95f6-4b40-85b6-b980d813e0cd None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:39.648024 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2adf72-95f6-4b40-85b6-b980d813e0cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:39.705323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:39.726707 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-1b2adf72-95f6-4b40-85b6-b980d813e0cd None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:39.727967 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1b2adf72-95f6-4b40-85b6-b980d813e0cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:39.955497 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 728/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 14:23:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:23:40.252528 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f881b3b3-4224-470f-8969-aea988e0604a None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:40.254083 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-f881b3b3-4224-470f-8969-aea988e0604a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:40.374762 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f881b3b3-4224-470f-8969-aea988e0604a None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:40.424808 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:40.464754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362b2719-429d-4e84-833f-80b7c7b76e76 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:40.465372 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362b2719-429d-4e84-833f-80b7c7b76e76 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:40.465654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362b2719-429d-4e84-833f-80b7c7b76e76 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:40.465972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362b2719-429d-4e84-833f-80b7c7b76e76 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:40.467587 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-362b2719-429d-4e84-833f-80b7c7b76e76 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:40.589477 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362b2719-429d-4e84-833f-80b7c7b76e76 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:40.590858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362b2719-429d-4e84-833f-80b7c7b76e76 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:40.611194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 721/1449] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:23:43.032705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d572ddf1-9885-4197-aa15-1fd5af7148c6 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:43.033449 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d572ddf1-9885-4197-aa15-1fd5af7148c6 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:43.033621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d572ddf1-9885-4197-aa15-1fd5af7148c6 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:43.035448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 729/1450] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 14:23:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:43.041651 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a23cf1a-e9de-4f8e-ad9f-709714ee5f83 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:43.042185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a23cf1a-e9de-4f8e-ad9f-709714ee5f83 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:43.042626 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a23cf1a-e9de-4f8e-ad9f-709714ee5f83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:43.120259 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-6a23cf1a-e9de-4f8e-ad9f-709714ee5f83 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:43.121540 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-6a23cf1a-e9de-4f8e-ad9f-709714ee5f83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:43.146072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:43.236774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 722/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 14:23:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:23:43.535477 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d572ddf1-9885-4197-aa15-1fd5af7148c6 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:43.537105 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d572ddf1-9885-4197-aa15-1fd5af7148c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:43.659543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d572ddf1-9885-4197-aa15-1fd5af7148c6 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:43.681703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:43.742151 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88d8335-ce78-4b3f-936c-537a9f7572fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:43.742658 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88d8335-ce78-4b3f-936c-537a9f7572fb service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:43.742927 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88d8335-ce78-4b3f-936c-537a9f7572fb service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:43.743193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88d8335-ce78-4b3f-936c-537a9f7572fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:43.744839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c88d8335-ce78-4b3f-936c-537a9f7572fb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:43.879304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88d8335-ce78-4b3f-936c-537a9f7572fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:43.880775 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88d8335-ce78-4b3f-936c-537a9f7572fb service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:43.903985 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 730/1452] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:23:46.235000 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c9e964-2d0e-45af-a1c3-cc38eccb546e None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:46.235590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c9e964-2d0e-45af-a1c3-cc38eccb546e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:46.235590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c9e964-2d0e-45af-a1c3-cc38eccb546e None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:46.237639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 723/1453] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 14:23:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:46.243619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7052c9-0bc2-4e5d-8ba1-109b4d56a32e None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:46.244553 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7052c9-0bc2-4e5d-8ba1-109b4d56a32e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:46.244553 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7052c9-0bc2-4e5d-8ba1-109b4d56a32e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:46.319170 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:46.323827 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-eb7052c9-0bc2-4e5d-8ba1-109b4d56a32e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:46.325278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-eb7052c9-0bc2-4e5d-8ba1-109b4d56a32e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:46.452869 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 731/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 14:23:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:23:46.744890 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c9e964-2d0e-45af-a1c3-cc38eccb546e None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:46.746433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-92c9e964-2d0e-45af-a1c3-cc38eccb546e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:46.856633 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c9e964-2d0e-45af-a1c3-cc38eccb546e None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:46.909342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:46.935711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e771c66-2633-4cc8-b368-19ad37678164 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:46.936333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e771c66-2633-4cc8-b368-19ad37678164 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:46.936639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e771c66-2633-4cc8-b368-19ad37678164 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:46.936914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e771c66-2633-4cc8-b368-19ad37678164 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:46.938525 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9e771c66-2633-4cc8-b368-19ad37678164 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:47.066198 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e771c66-2633-4cc8-b368-19ad37678164 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:47.067596 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e771c66-2633-4cc8-b368-19ad37678164 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:47.087682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 724/1455] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:23:49.399472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d680765f-dfd7-471a-966a-9d0e306000cc None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:49.400083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d680765f-dfd7-471a-966a-9d0e306000cc None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:49.400199 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d680765f-dfd7-471a-966a-9d0e306000cc None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:49.401652 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 732/1456] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 14:23:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:49.407622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32393c28-50d1-43b3-a715-0aa572cfcef8 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:49.408064 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32393c28-50d1-43b3-a715-0aa572cfcef8 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:49.408342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32393c28-50d1-43b3-a715-0aa572cfcef8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:49.480765 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-32393c28-50d1-43b3-a715-0aa572cfcef8 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:49.482019 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-32393c28-50d1-43b3-a715-0aa572cfcef8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:49.514227 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:49.588612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 725/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 14:23:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:23:49.883162 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d680765f-dfd7-471a-966a-9d0e306000cc None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:49.884726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d680765f-dfd7-471a-966a-9d0e306000cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:50.003984 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d680765f-dfd7-471a-966a-9d0e306000cc None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:50.035640 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:50.085189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ac67f1-4e9b-4041-8e85-e13d193c34ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:50.085789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ac67f1-4e9b-4041-8e85-e13d193c34ae service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:50.086072 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ac67f1-4e9b-4041-8e85-e13d193c34ae service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:50.086462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ac67f1-4e9b-4041-8e85-e13d193c34ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:50.088437 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-08ac67f1-4e9b-4041-8e85-e13d193c34ae service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:50.210145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ac67f1-4e9b-4041-8e85-e13d193c34ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:50.211469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ac67f1-4e9b-4041-8e85-e13d193c34ae service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:50.230849 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 733/1458] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:23:52.538619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc5c771-a811-4fe8-be45-984297125532 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:52.539153 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc5c771-a811-4fe8-be45-984297125532 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:52.539153 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc5c771-a811-4fe8-be45-984297125532 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:52.540521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 726/1459] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 14:23:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:52.545769 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b8b3b3-3ee5-4219-892f-4ac8e5213592 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:52.546180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b8b3b3-3ee5-4219-892f-4ac8e5213592 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:52.546310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b8b3b3-3ee5-4219-892f-4ac8e5213592 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:52.614511 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-87b8b3b3-3ee5-4219-892f-4ac8e5213592 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:52.615728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-87b8b3b3-3ee5-4219-892f-4ac8e5213592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:52.624182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:52.720625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 734/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 14:23:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:23:53.005523 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc5c771-a811-4fe8-be45-984297125532 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:53.007054 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7dc5c771-a811-4fe8-be45-984297125532 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:53.123421 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc5c771-a811-4fe8-be45-984297125532 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:53.181381 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:53.203102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340359bb-d870-4125-a4cb-bab67c21c8e0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:53.203709 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340359bb-d870-4125-a4cb-bab67c21c8e0 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:53.203990 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340359bb-d870-4125-a4cb-bab67c21c8e0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:53.204330 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340359bb-d870-4125-a4cb-bab67c21c8e0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:53.206291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-340359bb-d870-4125-a4cb-bab67c21c8e0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:53.318332 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340359bb-d870-4125-a4cb-bab67c21c8e0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:53.319679 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340359bb-d870-4125-a4cb-bab67c21c8e0 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:53.338142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 727/1461] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:23:55.581024 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79410d3d-578c-498f-9529-2ec547d437e4 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:55.581625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79410d3d-578c-498f-9529-2ec547d437e4 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:55.581625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79410d3d-578c-498f-9529-2ec547d437e4 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:55.582892 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 735/1462] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 14:23:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:55.588591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7022943e-13d9-4b57-a0fd-90d8f2f6823d None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:55.589074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7022943e-13d9-4b57-a0fd-90d8f2f6823d None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:55.589172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7022943e-13d9-4b57-a0fd-90d8f2f6823d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:55.656894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-7022943e-13d9-4b57-a0fd-90d8f2f6823d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:55.658155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7022943e-13d9-4b57-a0fd-90d8f2f6823d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:55.702719 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:55.769732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 728/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 14:23:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:23:56.065468 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79410d3d-578c-498f-9529-2ec547d437e4 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:56.066995 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-79410d3d-578c-498f-9529-2ec547d437e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:56.173193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79410d3d-578c-498f-9529-2ec547d437e4 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:56.215731 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:56.253001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6331203-a24d-4948-b5b3-f8f7f94938f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:56.253603 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6331203-a24d-4948-b5b3-f8f7f94938f5 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:56.253885 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6331203-a24d-4948-b5b3-f8f7f94938f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:56.254146 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6331203-a24d-4948-b5b3-f8f7f94938f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:56.255743 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-e6331203-a24d-4948-b5b3-f8f7f94938f5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:56.368674 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6331203-a24d-4948-b5b3-f8f7f94938f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:56.370105 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6331203-a24d-4948-b5b3-f8f7f94938f5 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:56.388749 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 736/1464] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:23:58.647414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c97b4c-a291-4bd6-87af-2dc049908654 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:58.647951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c97b4c-a291-4bd6-87af-2dc049908654 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:58.647951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c97b4c-a291-4bd6-87af-2dc049908654 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:58.649342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 729/1465] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 14:23:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:23:58.654765 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ee5e7b-2430-4c15-b803-34a13c871fea None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:58.655295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ee5e7b-2430-4c15-b803-34a13c871fea None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:58.655295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ee5e7b-2430-4c15-b803-34a13c871fea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:58.723556 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-d1ee5e7b-2430-4c15-b803-34a13c871fea None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:23:58.724795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d1ee5e7b-2430-4c15-b803-34a13c871fea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:58.742602 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:58.822635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 737/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 14:23:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:23:59.107022 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c97b4c-a291-4bd6-87af-2dc049908654 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:23:59.108649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-63c97b4c-a291-4bd6-87af-2dc049908654 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:59.223625 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c97b4c-a291-4bd6-87af-2dc049908654 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:23:59.291985 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:23:59.300748 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41200672-891d-404c-a6a1-9143fdf8b198 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:23:59.301256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41200672-891d-404c-a6a1-9143fdf8b198 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:23:59.301602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41200672-891d-404c-a6a1-9143fdf8b198 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:23:59.301946 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41200672-891d-404c-a6a1-9143fdf8b198 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:23:59.303476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-41200672-891d-404c-a6a1-9143fdf8b198 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:23:59.426536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41200672-891d-404c-a6a1-9143fdf8b198 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:23:59.428004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41200672-891d-404c-a6a1-9143fdf8b198 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:23:59.447284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 730/1467] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:23:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:24:01.659568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632301c2-5dae-4f9e-812b-9d993d4bbe6c None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:01.660152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632301c2-5dae-4f9e-812b-9d993d4bbe6c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:01.660152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632301c2-5dae-4f9e-812b-9d993d4bbe6c None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:01.661391 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 738/1468] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 14:24:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:24:01.667462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0317a53-d33a-41d9-ac47-e3259da1339c None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:01.668012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0317a53-d33a-41d9-ac47-e3259da1339c None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:01.668012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0317a53-d33a-41d9-ac47-e3259da1339c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:01.737964 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-c0317a53-d33a-41d9-ac47-e3259da1339c None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:24:01.739200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c0317a53-d33a-41d9-ac47-e3259da1339c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:01.796696 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:01.848174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 731/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 14:24:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:24:02.135836 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632301c2-5dae-4f9e-812b-9d993d4bbe6c None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:24:02.137381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-632301c2-5dae-4f9e-812b-9d993d4bbe6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:02.243076 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632301c2-5dae-4f9e-812b-9d993d4bbe6c None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:24:02.290730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:02.318692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9079b735-27c8-4b8e-b9d1-f9ae2f5d9581 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:24:02.319286 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9079b735-27c8-4b8e-b9d1-f9ae2f5d9581 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:02.319563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9079b735-27c8-4b8e-b9d1-f9ae2f5d9581 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:02.319829 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9079b735-27c8-4b8e-b9d1-f9ae2f5d9581 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:02.321440 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9079b735-27c8-4b8e-b9d1-f9ae2f5d9581 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:02.435977 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9079b735-27c8-4b8e-b9d1-f9ae2f5d9581 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:24:02.437360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9079b735-27c8-4b8e-b9d1-f9ae2f5d9581 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:24:02.463178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 739/1470] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:24:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:24:11.846702 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=22 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:12.454768 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=66 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:23.699823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50e7b23-839c-47d0-9910-90531de2e67a None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:23.700634 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50e7b23-839c-47d0-9910-90531de2e67a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:23.700634 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50e7b23-839c-47d0-9910-90531de2e67a None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:23.702266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 732/1471] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:24:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:24:23.708931 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28b1e60-d1d7-4b02-ad88-9e0264852d3f None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:23.709782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28b1e60-d1d7-4b02-ad88-9e0264852d3f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:23.709782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28b1e60-d1d7-4b02-ad88-9e0264852d3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:23.785801 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-f28b1e60-d1d7-4b02-ad88-9e0264852d3f None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:24:23.787027 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f28b1e60-d1d7-4b02-ad88-9e0264852d3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:23.903009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 740/1472] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:24:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:24:24.223766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfc2a8a-39ef-41ff-8a9b-97a037d39e4f None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:24.224102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfc2a8a-39ef-41ff-8a9b-97a037d39e4f None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:24.224473 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfc2a8a-39ef-41ff-8a9b-97a037d39e4f None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:24.225852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 733/1473] 10.222.0.64 () {54 vars in 962 bytes} [Tue Oct 4 14:24:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:24:24.231475 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e2da3b-f5cb-4b51-a4ce-f03779351407 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:24.231785 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e2da3b-f5cb-4b51-a4ce-f03779351407 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:24.232173 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e2da3b-f5cb-4b51-a4ce-f03779351407 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:24.333447 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-57e2da3b-f5cb-4b51-a4ce-f03779351407 None None] MFA Rules not processed for user `35a5a4ef40204e158649ca25583bbde7`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:24:24.335572 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-57e2da3b-f5cb-4b51-a4ce-f03779351407 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:24.378004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:24.449762 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 741/1474] 10.222.0.64 () {58 vars in 1059 bytes} [Tue Oct 4 14:24:24 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:24:24.460165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfc2a8a-39ef-41ff-8a9b-97a037d39e4f None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:24:24.461803 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-3bfc2a8a-39ef-41ff-8a9b-97a037d39e4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:24.590134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfc2a8a-39ef-41ff-8a9b-97a037d39e4f None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:24:24.695583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ef8f2e-6691-4edb-bf91-970bdfdd9e5e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35a5a4ef40204e158649ca25583bbde7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:24:24.696257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ef8f2e-6691-4edb-bf91-970bdfdd9e5e service glance] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:24.696628 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ef8f2e-6691-4edb-bf91-970bdfdd9e5e service glance] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:24.696955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ef8f2e-6691-4edb-bf91-970bdfdd9e5e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:24.698724 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-07ef8f2e-6691-4edb-bf91-970bdfdd9e5e service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:24.766801 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:24.843199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ef8f2e-6691-4edb-bf91-970bdfdd9e5e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:24:24.845389 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ef8f2e-6691-4edb-bf91-970bdfdd9e5e service glance] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:24:24.868270 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 734/1475] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:24:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:24:27.298826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d148b978-8df2-4313-b299-ca76c6ca8d7c None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:27.299436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d148b978-8df2-4313-b299-ca76c6ca8d7c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:27.299436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d148b978-8df2-4313-b299-ca76c6ca8d7c None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:27.300821 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 742/1476] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:24:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:24:27.307070 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fffe07-ee5d-4f26-ae9c-c9ecf416ef01 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:27.307960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fffe07-ee5d-4f26-ae9c-c9ecf416ef01 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:27.307960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fffe07-ee5d-4f26-ae9c-c9ecf416ef01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:27.404924 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-24fffe07-ee5d-4f26-ae9c-c9ecf416ef01 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:24:27.406885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-24fffe07-ee5d-4f26-ae9c-c9ecf416ef01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:27.579963 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 735/1477] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:24:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:24:27.864893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d148b978-8df2-4313-b299-ca76c6ca8d7c None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:24:27.867735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d148b978-8df2-4313-b299-ca76c6ca8d7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:28.023508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d148b978-8df2-4313-b299-ca76c6ca8d7c None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:24:28.081836 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:28.132829 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e96642-39ef-421e-b169-227b579657f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:24:28.133350 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e96642-39ef-421e-b169-227b579657f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:28.133627 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e96642-39ef-421e-b169-227b579657f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:28.133897 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e96642-39ef-421e-b169-227b579657f7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:28.134539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e96642-39ef-421e-b169-227b579657f7 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:24:28.136343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: WARNING py.warnings [None req-10e96642-39ef-421e-b169-227b579657f7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 14:24:28.136343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: warnings.warn(msg) Oct 04 14:24:28.136343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: Oct 04 14:24:28.137495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e96642-39ef-421e-b169-227b579657f7 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:24:28.159434 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 743/1478] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Oct 4 14:24:27 2022] GET /identity/v3/projects => generated 2553 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:24:30.505640 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a872f98-9149-4ad7-984f-875d791df81c None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:30.506265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a872f98-9149-4ad7-984f-875d791df81c None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:30.506265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a872f98-9149-4ad7-984f-875d791df81c None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:30.507539 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 736/1479] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:24:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:24:30.513817 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c76542b-4a86-45b5-bfb6-8cc964203da7 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:30.514573 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c76542b-4a86-45b5-bfb6-8cc964203da7 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:30.514573 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c76542b-4a86-45b5-bfb6-8cc964203da7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:30.592292 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-6c76542b-4a86-45b5-bfb6-8cc964203da7 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:24:30.593575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6c76542b-4a86-45b5-bfb6-8cc964203da7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:30.707106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 744/1480] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:24:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:24:31.008668 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a872f98-9149-4ad7-984f-875d791df81c None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:24:31.010381 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-4a872f98-9149-4ad7-984f-875d791df81c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:31.022294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:31.204368 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a872f98-9149-4ad7-984f-875d791df81c None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:24:31.308437 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:31.345039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b89127-0e4f-4d95-a671-a62a1774934b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:24:31.345857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b89127-0e4f-4d95-a671-a62a1774934b service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:31.346144 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b89127-0e4f-4d95-a671-a62a1774934b service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:31.346412 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b89127-0e4f-4d95-a671-a62a1774934b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:31.348111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d8b89127-0e4f-4d95-a671-a62a1774934b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:31.451850 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b89127-0e4f-4d95-a671-a62a1774934b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:24:31.455644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b89127-0e4f-4d95-a671-a62a1774934b service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:24:31.476694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 737/1481] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:24:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 469 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:24:34.011870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514369e0-3db2-438f-805f-8284a6e08d7a None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:34.012447 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514369e0-3db2-438f-805f-8284a6e08d7a None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:34.012447 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514369e0-3db2-438f-805f-8284a6e08d7a None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:34.013725 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 745/1482] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:24:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:24:34.020631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c335b10c-13aa-4fd9-afae-84f1772d0ad9 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:34.021236 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c335b10c-13aa-4fd9-afae-84f1772d0ad9 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:34.021236 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c335b10c-13aa-4fd9-afae-84f1772d0ad9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:34.100352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-c335b10c-13aa-4fd9-afae-84f1772d0ad9 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:24:34.101636 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c335b10c-13aa-4fd9-afae-84f1772d0ad9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:34.154569 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:34.212749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 738/1483] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:24:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:24:34.506323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514369e0-3db2-438f-805f-8284a6e08d7a None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:24:34.507864 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-514369e0-3db2-438f-805f-8284a6e08d7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:34.534602 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:34.631421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514369e0-3db2-438f-805f-8284a6e08d7a None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:24:34.718572 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4c0b32-5701-467d-8c2a-84fa8e715ac7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:24:34.719134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4c0b32-5701-467d-8c2a-84fa8e715ac7 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:34.720153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4c0b32-5701-467d-8c2a-84fa8e715ac7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:34.720469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4c0b32-5701-467d-8c2a-84fa8e715ac7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:34.722377 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9f4c0b32-5701-467d-8c2a-84fa8e715ac7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:34.805684 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:34.845139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4c0b32-5701-467d-8c2a-84fa8e715ac7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:24:34.846578 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4c0b32-5701-467d-8c2a-84fa8e715ac7 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:24:34.865697 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 746/1484] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:24:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:24:37.149745 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93097178-f1f6-4abb-acb1-ce5a1ab64662 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:37.150282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93097178-f1f6-4abb-acb1-ce5a1ab64662 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:37.150282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93097178-f1f6-4abb-acb1-ce5a1ab64662 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:37.151652 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 739/1485] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:24:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:24:37.157683 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b023493b-210b-46e1-9014-97227e3f50a4 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:37.158226 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b023493b-210b-46e1-9014-97227e3f50a4 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:37.158226 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b023493b-210b-46e1-9014-97227e3f50a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:37.229629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-b023493b-210b-46e1-9014-97227e3f50a4 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:24:37.230852 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b023493b-210b-46e1-9014-97227e3f50a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:37.336398 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 747/1486] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:24:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:24:37.629211 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93097178-f1f6-4abb-acb1-ce5a1ab64662 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:24:37.630763 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-93097178-f1f6-4abb-acb1-ce5a1ab64662 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:37.749404 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93097178-f1f6-4abb-acb1-ce5a1ab64662 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:24:37.822302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:37.823823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8d89cf-2519-4a07-97e7-ecd37af47eb7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:24:37.824500 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8d89cf-2519-4a07-97e7-ecd37af47eb7 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:37.824973 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8d89cf-2519-4a07-97e7-ecd37af47eb7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:37.825414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8d89cf-2519-4a07-97e7-ecd37af47eb7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:37.827222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-cc8d89cf-2519-4a07-97e7-ecd37af47eb7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:37.937955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8d89cf-2519-4a07-97e7-ecd37af47eb7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:24:37.939385 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8d89cf-2519-4a07-97e7-ecd37af47eb7 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:24:37.957615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 740/1487] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:24:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:24:40.324199 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5a0e33-9670-436b-88bf-eb169cc9c57b None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:40.324803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5a0e33-9670-436b-88bf-eb169cc9c57b None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:40.324803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5a0e33-9670-436b-88bf-eb169cc9c57b None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:40.326138 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 748/1488] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:24:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:24:40.332255 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc2374c-f2ed-4dd2-9a57-d68f826c5ddc None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:40.332732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc2374c-f2ed-4dd2-9a57-d68f826c5ddc None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:40.332865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc2374c-f2ed-4dd2-9a57-d68f826c5ddc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:40.402012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-abc2374c-f2ed-4dd2-9a57-d68f826c5ddc None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:24:40.403236 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-abc2374c-f2ed-4dd2-9a57-d68f826c5ddc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:40.459124 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:40.506306 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 741/1489] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:24:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:24:40.798463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5a0e33-9670-436b-88bf-eb169cc9c57b None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:24:40.800003 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-fe5a0e33-9670-436b-88bf-eb169cc9c57b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:40.839874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:40.911459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5a0e33-9670-436b-88bf-eb169cc9c57b None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:24:40.986807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17263199-6433-42ca-a816-bd1bd5c38551 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:24:40.987504 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17263199-6433-42ca-a816-bd1bd5c38551 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:40.987793 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17263199-6433-42ca-a816-bd1bd5c38551 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:40.988069 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17263199-6433-42ca-a816-bd1bd5c38551 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:40.989687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-17263199-6433-42ca-a816-bd1bd5c38551 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:41.070408 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:41.100894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17263199-6433-42ca-a816-bd1bd5c38551 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:24:41.102591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17263199-6433-42ca-a816-bd1bd5c38551 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:24:41.121629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 749/1490] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:24:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:24:43.543898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d516215-79da-4d2d-a3cf-97022cbf2a76 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:43.544492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d516215-79da-4d2d-a3cf-97022cbf2a76 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:43.544492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d516215-79da-4d2d-a3cf-97022cbf2a76 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:43.545838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 742/1491] 10.222.0.64 () {54 vars in 927 bytes} [Tue Oct 4 14:24:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:24:43.555870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b2318d-a4be-4464-821c-fd2a594d8e74 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:43.558305 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b2318d-a4be-4464-821c-fd2a594d8e74 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:43.558305 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b2318d-a4be-4464-821c-fd2a594d8e74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:43.624646 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-36b2318d-a4be-4464-821c-fd2a594d8e74 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:24:43.626455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-36b2318d-a4be-4464-821c-fd2a594d8e74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:43.724351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 750/1492] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Oct 4 14:24:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:24:44.016193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d516215-79da-4d2d-a3cf-97022cbf2a76 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:24:44.017766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8d516215-79da-4d2d-a3cf-97022cbf2a76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:44.129778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d516215-79da-4d2d-a3cf-97022cbf2a76 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:24:44.209658 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244b5ac0-0752-49b1-bd2e-930d0e968891 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:24:44.210271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244b5ac0-0752-49b1-bd2e-930d0e968891 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:24:44.210552 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244b5ac0-0752-49b1-bd2e-930d0e968891 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:24:44.210831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244b5ac0-0752-49b1-bd2e-930d0e968891 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:24:44.212486 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-244b5ac0-0752-49b1-bd2e-930d0e968891 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:24:44.222276 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:44.327085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244b5ac0-0752-49b1-bd2e-930d0e968891 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:24:44.329099 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244b5ac0-0752-49b1-bd2e-930d0e968891 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:24:44.347739 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 743/1493] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:24:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:24:53.722788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=82 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:24:54.345470 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=50 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:26:01.591230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313fe96c-356d-4055-b8f6-fac8796aaca8 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:26:01.591884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313fe96c-356d-4055-b8f6-fac8796aaca8 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:26:01.591884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313fe96c-356d-4055-b8f6-fac8796aaca8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:26:01.684679 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-313fe96c-356d-4055-b8f6-fac8796aaca8 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:26:01.685918 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-313fe96c-356d-4055-b8f6-fac8796aaca8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:26:01.803237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 751/1494] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:26:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:26:01.812139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4161e497-1df6-4d3e-86a5-fb5e6889e183 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:26:01.812869 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4161e497-1df6-4d3e-86a5-fb5e6889e183 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:26:01.812869 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4161e497-1df6-4d3e-86a5-fb5e6889e183 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:26:01.935801 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-4161e497-1df6-4d3e-86a5-fb5e6889e183 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:26:01.938467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-4161e497-1df6-4d3e-86a5-fb5e6889e183 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:26:02.140536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 744/1495] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:26:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 330 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:26:02.147532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313fe96c-356d-4055-b8f6-fac8796aaca8 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:26:02.149015 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-313fe96c-356d-4055-b8f6-fac8796aaca8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:26:02.162470 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313fe96c-356d-4055-b8f6-fac8796aaca8 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:26:02.243105 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:26:02.286321 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abb4567-bdc2-43cb-9bce-d930fbaaaf7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:26:02.286938 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abb4567-bdc2-43cb-9bce-d930fbaaaf7e admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:26:02.287218 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abb4567-bdc2-43cb-9bce-d930fbaaaf7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:26:02.287488 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abb4567-bdc2-43cb-9bce-d930fbaaaf7e admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:26:02.288150 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abb4567-bdc2-43cb-9bce-d930fbaaaf7e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:26:02.288489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abb4567-bdc2-43cb-9bce-d930fbaaaf7e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:26:02.289533 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: WARNING py.warnings [None req-2abb4567-bdc2-43cb-9bce-d930fbaaaf7e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 04 14:26:02.289533 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: warnings.warn(msg) Oct 04 14:26:02.289533 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: Oct 04 14:26:02.290441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abb4567-bdc2-43cb-9bce-d930fbaaaf7e admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:26:02.304543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 752/1496] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:26:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:26:02.314423 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4161e497-1df6-4d3e-86a5-fb5e6889e183 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:26:02.325096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4161e497-1df6-4d3e-86a5-fb5e6889e183 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:26:02.330771 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933e6990-3d21-4ee5-b986-c1c9e11417bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:26:02.331461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933e6990-3d21-4ee5-b986-c1c9e11417bb admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:26:02.331796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933e6990-3d21-4ee5-b986-c1c9e11417bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:26:02.332091 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933e6990-3d21-4ee5-b986-c1c9e11417bb admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:26:02.332970 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933e6990-3d21-4ee5-b986-c1c9e11417bb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:26:02.335109 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: WARNING py.warnings [None req-933e6990-3d21-4ee5-b986-c1c9e11417bb admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 14:26:02.335109 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: warnings.warn(msg) Oct 04 14:26:02.335109 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: Oct 04 14:26:02.336512 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933e6990-3d21-4ee5-b986-c1c9e11417bb admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:26:02.383707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 745/1497] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:26:02 2022] POST /identity/v3/projects => generated 379 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:26:02.385827 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abb4567-bdc2-43cb-9bce-d930fbaaaf7e admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:26:02.398366 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abb4567-bdc2-43cb-9bce-d930fbaaaf7e admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:26:02.495389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b74ab2-c76e-409c-a463-4a3fdead1660 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:26:02.496486 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b74ab2-c76e-409c-a463-4a3fdead1660 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:26:02.496990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b74ab2-c76e-409c-a463-4a3fdead1660 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:26:02.497470 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b74ab2-c76e-409c-a463-4a3fdead1660 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:26:02.499173 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b74ab2-c76e-409c-a463-4a3fdead1660 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:26:02.501159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: WARNING py.warnings [None req-89b74ab2-c76e-409c-a463-4a3fdead1660 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 14:26:02.501159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: warnings.warn(msg) Oct 04 14:26:02.501159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: Oct 04 14:26:02.503476 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b74ab2-c76e-409c-a463-4a3fdead1660 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:26:02.533567 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 753/1498] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:26:02 2022] POST /identity/v3/users => generated 290 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:26:02.539810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933e6990-3d21-4ee5-b986-c1c9e11417bb admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:26:02.552804 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933e6990-3d21-4ee5-b986-c1c9e11417bb admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:26:02.561457 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381326bb-92cf-4542-b6c8-e3af6797619e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:26:02.562653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381326bb-92cf-4542-b6c8-e3af6797619e admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:26:02.563113 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381326bb-92cf-4542-b6c8-e3af6797619e admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:26:02.563563 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381326bb-92cf-4542-b6c8-e3af6797619e admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:26:02.564733 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381326bb-92cf-4542-b6c8-e3af6797619e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:26:02.566460 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: WARNING py.warnings [None req-381326bb-92cf-4542-b6c8-e3af6797619e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 04 14:26:02.566460 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: warnings.warn(msg) Oct 04 14:26:02.566460 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: Oct 04 14:26:02.568106 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381326bb-92cf-4542-b6c8-e3af6797619e admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:26:02.595332 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 746/1499] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:26:02 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:26:02.599721 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b74ab2-c76e-409c-a463-4a3fdead1660 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:26:02.611289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b74ab2-c76e-409c-a463-4a3fdead1660 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:26:02.616581 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630774bf-5c4d-4414-9c38-b555d515ee48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:26:02.617208 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630774bf-5c4d-4414-9c38-b555d515ee48 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:26:02.617480 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630774bf-5c4d-4414-9c38-b555d515ee48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:26:02.617748 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630774bf-5c4d-4414-9c38-b555d515ee48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:26:02.618582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630774bf-5c4d-4414-9c38-b555d515ee48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:26:02.619535 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: WARNING py.warnings [None req-630774bf-5c4d-4414-9c38-b555d515ee48 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 04 14:26:02.619535 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: warnings.warn(msg) Oct 04 14:26:02.619535 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: Oct 04 14:26:02.620465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630774bf-5c4d-4414-9c38-b555d515ee48 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:26:02.636500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 754/1500] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:26:02 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:26:02.641812 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381326bb-92cf-4542-b6c8-e3af6797619e admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:26:02.652838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381326bb-92cf-4542-b6c8-e3af6797619e admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:26:02.658930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0386fd6d-4493-472d-9b3b-e72ce26fed28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:26:02.659705 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0386fd6d-4493-472d-9b3b-e72ce26fed28 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:26:02.660038 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0386fd6d-4493-472d-9b3b-e72ce26fed28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:26:02.660515 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0386fd6d-4493-472d-9b3b-e72ce26fed28 admin admin] PATH_INFO: `/v3/projects/ebc1b4a903604a7f8112514f272da4c2/users/40f7b673dcee488a86a496607bb747d4/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:26:02.679662 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0386fd6d-4493-472d-9b3b-e72ce26fed28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ebc1b4a903604a7f8112514f272da4c2, user_id=40f7b673dcee488a86a496607bb747d4, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:26:02.680832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: WARNING py.warnings [None req-0386fd6d-4493-472d-9b3b-e72ce26fed28 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 14:26:02.680832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: warnings.warn(msg) Oct 04 14:26:02.680832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: Oct 04 14:26:02.682769 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0386fd6d-4493-472d-9b3b-e72ce26fed28 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:26:02.698631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 747/1501] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:26:02 2022] PUT /identity/v3/projects/ebc1b4a903604a7f8112514f272da4c2/users/40f7b673dcee488a86a496607bb747d4/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:26:02.712096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241b29b4-d5f2-483b-be18-36f0b697c029 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:26:02.712582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241b29b4-d5f2-483b-be18-36f0b697c029 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:26:02.712901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241b29b4-d5f2-483b-be18-36f0b697c029 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:26:02.757746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-241b29b4-d5f2-483b-be18-36f0b697c029 None None] MFA Rules not processed for user `40f7b673dcee488a86a496607bb747d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:26:02.759051 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-241b29b4-d5f2-483b-be18-36f0b697c029 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:26:02.841018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 755/1502] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:26:02 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:26:02.856688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c041e72f-a40b-4b6a-976b-1f415ee803d9 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:26:02.857317 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c041e72f-a40b-4b6a-976b-1f415ee803d9 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:26:02.857317 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c041e72f-a40b-4b6a-976b-1f415ee803d9 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:26:02.860045 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 748/1503] 10.222.0.64 () {54 vars in 963 bytes} [Tue Oct 4 14:26:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:26:02.866116 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63999dc8-d353-4f8c-bdd1-6fd1845d4977 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:26:02.866445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63999dc8-d353-4f8c-bdd1-6fd1845d4977 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:26:02.866650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63999dc8-d353-4f8c-bdd1-6fd1845d4977 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:26:02.900915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:26:02.948245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-63999dc8-d353-4f8c-bdd1-6fd1845d4977 None None] MFA Rules not processed for user `058c027ade94498fbab7b9347e87c9c3`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:26:02.949581 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-63999dc8-d353-4f8c-bdd1-6fd1845d4977 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:26:03.047435 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 756/1504] 10.222.0.64 () {58 vars in 1060 bytes} [Tue Oct 4 14:26:02 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:26:03.058058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c041e72f-a40b-4b6a-976b-1f415ee803d9 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:26:03.063888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:26:03.071572 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c041e72f-a40b-4b6a-976b-1f415ee803d9 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:26:03.076278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db01ed17-7f90-440f-9f12-68cb88a974a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:26:03.077453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db01ed17-7f90-440f-9f12-68cb88a974a2 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:26:03.077737 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db01ed17-7f90-440f-9f12-68cb88a974a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:26:03.078065 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db01ed17-7f90-440f-9f12-68cb88a974a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:26:03.087820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db01ed17-7f90-440f-9f12-68cb88a974a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:26:03.089369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db01ed17-7f90-440f-9f12-68cb88a974a2 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:26:03.100620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 749/1505] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:26:03 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:26:03.153092 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b5fb69-bc89-4183-9bb3-870db59016f4 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:26:03.153497 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b5fb69-bc89-4183-9bb3-870db59016f4 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:26:03.153938 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b5fb69-bc89-4183-9bb3-870db59016f4 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:26:03.156097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 757/1506] 10.222.0.64 () {52 vars in 830 bytes} [Tue Oct 4 14:26:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:26:03.160883 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db01ed17-7f90-440f-9f12-68cb88a974a2 service cinder] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:26:03.238874 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db01ed17-7f90-440f-9f12-68cb88a974a2 service cinder] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:26:03.243896 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df30ee01-c480-4ab5-8ee8-bed7a0751281 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:26:03.244585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df30ee01-c480-4ab5-8ee8-bed7a0751281 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:26:03.244882 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df30ee01-c480-4ab5-8ee8-bed7a0751281 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:26:03.245160 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df30ee01-c480-4ab5-8ee8-bed7a0751281 admin admin] PATH_INFO: `/v3/users/40f7b673dcee488a86a496607bb747d4` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:26:03.246615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df30ee01-c480-4ab5-8ee8-bed7a0751281 admin admin] RBAC: Authorizing `identity:delete_user(user_id=40f7b673dcee488a86a496607bb747d4)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:26:03.247788 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: WARNING py.warnings [None req-df30ee01-c480-4ab5-8ee8-bed7a0751281 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 14:26:03.247788 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: warnings.warn(msg) Oct 04 14:26:03.247788 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: Oct 04 14:26:03.250001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df30ee01-c480-4ab5-8ee8-bed7a0751281 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:26:03.335298 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-df30ee01-c480-4ab5-8ee8-bed7a0751281 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '40f7b673dcee488a86a496607bb747d4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:26:03.336060 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-df30ee01-c480-4ab5-8ee8-bed7a0751281 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '40f7b673dcee488a86a496607bb747d4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:26:03.336915 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-df30ee01-c480-4ab5-8ee8-bed7a0751281 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '40f7b673dcee488a86a496607bb747d4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:26:03.349726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-df30ee01-c480-4ab5-8ee8-bed7a0751281 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '40f7b673dcee488a86a496607bb747d4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:26:03.378193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-df30ee01-c480-4ab5-8ee8-bed7a0751281 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '40f7b673dcee488a86a496607bb747d4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:26:03.402717 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 750/1507] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:26:03 2022] DELETE /identity/v3/users/40f7b673dcee488a86a496607bb747d4 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:26:03.407383 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee7fca2-d4a0-4bc1-9d1a-b2877f2f1a65 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:26:03.407699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee7fca2-d4a0-4bc1-9d1a-b2877f2f1a65 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:26:03.408186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee7fca2-d4a0-4bc1-9d1a-b2877f2f1a65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:26:03.428929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-4ee7fca2-d4a0-4bc1-9d1a-b2877f2f1a65 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:26:03.430148 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-4ee7fca2-d4a0-4bc1-9d1a-b2877f2f1a65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:26:03.497591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 758/1508] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:26:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:26:03.507996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df30ee01-c480-4ab5-8ee8-bed7a0751281 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:26:03.509592 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-df30ee01-c480-4ab5-8ee8-bed7a0751281 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:26:03.601249 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:26:03.608849 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df30ee01-c480-4ab5-8ee8-bed7a0751281 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:26:03.615259 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afe360d-79a3-408d-8497-6e95facfe278 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:26:03.618034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afe360d-79a3-408d-8497-6e95facfe278 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:26:03.618034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afe360d-79a3-408d-8497-6e95facfe278 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:26:03.618034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afe360d-79a3-408d-8497-6e95facfe278 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:26:03.633300 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afe360d-79a3-408d-8497-6e95facfe278 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:26:03.636094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afe360d-79a3-408d-8497-6e95facfe278 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:26:03.655786 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 751/1509] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:26:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:26:04.079464 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee7fca2-d4a0-4bc1-9d1a-b2877f2f1a65 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:26:04.081211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-4ee7fca2-d4a0-4bc1-9d1a-b2877f2f1a65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:26:04.093821 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee7fca2-d4a0-4bc1-9d1a-b2877f2f1a65 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:26:04.098554 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdec1450-6864-46be-8770-3d059ceb7682 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:26:04.099210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdec1450-6864-46be-8770-3d059ceb7682 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:26:04.099483 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdec1450-6864-46be-8770-3d059ceb7682 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:26:04.099750 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdec1450-6864-46be-8770-3d059ceb7682 admin admin] PATH_INFO: `/v3/projects/ebc1b4a903604a7f8112514f272da4c2` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:26:04.101513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdec1450-6864-46be-8770-3d059ceb7682 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ebc1b4a903604a7f8112514f272da4c2)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:26:04.103287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: WARNING py.warnings [None req-bdec1450-6864-46be-8770-3d059ceb7682 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 14:26:04.103287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: warnings.warn(msg) Oct 04 14:26:04.103287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: Oct 04 14:26:04.105516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdec1450-6864-46be-8770-3d059ceb7682 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:26:04.197358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bdec1450-6864-46be-8770-3d059ceb7682 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ebc1b4a903604a7f8112514f272da4c2', 'request_id': 'req-bdec1450-6864-46be-8770-3d059ceb7682'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:26:04.214432 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bdec1450-6864-46be-8770-3d059ceb7682 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ebc1b4a903604a7f8112514f272da4c2', 'request_id': 'req-bdec1450-6864-46be-8770-3d059ceb7682'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:26:04.223978 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bdec1450-6864-46be-8770-3d059ceb7682 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ebc1b4a903604a7f8112514f272da4c2', 'request_id': 'req-bdec1450-6864-46be-8770-3d059ceb7682'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:26:04.230692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:26:04.244260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bdec1450-6864-46be-8770-3d059ceb7682 admin admin] The token cache is being invalidate because project ebc1b4a903604a7f8112514f272da4c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:26:04.245069 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bdec1450-6864-46be-8770-3d059ceb7682 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:26:04.246296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bdec1450-6864-46be-8770-3d059ceb7682 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:26:04.248153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 759/1510] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:26:04 2022] DELETE /identity/v3/projects/ebc1b4a903604a7f8112514f272da4c2 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:26:13.646880 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:26:14.246509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:27:32.535239 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02dd1c3-67aa-43fb-ba41-55c16fef659f None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:32.536295 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02dd1c3-67aa-43fb-ba41-55c16fef659f None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:32.536295 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02dd1c3-67aa-43fb-ba41-55c16fef659f None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:32.537704 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 752/1511] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 14:27:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:27:32.544837 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58bcf12-5ab9-4aab-8f84-2bc585238b34 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:32.545621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58bcf12-5ab9-4aab-8f84-2bc585238b34 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:32.545621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58bcf12-5ab9-4aab-8f84-2bc585238b34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:32.566009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-a58bcf12-5ab9-4aab-8f84-2bc585238b34 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:27:32.567219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a58bcf12-5ab9-4aab-8f84-2bc585238b34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:27:32.643233 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 760/1512] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 14:27:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:27:32.805448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02dd1c3-67aa-43fb-ba41-55c16fef659f None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:27:32.807299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a02dd1c3-67aa-43fb-ba41-55c16fef659f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:27:32.929341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02dd1c3-67aa-43fb-ba41-55c16fef659f None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:27:32.936080 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2503bcab-6ad9-4398-a4ac-7baae4c1f1ca service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:27:32.936908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2503bcab-6ad9-4398-a4ac-7baae4c1f1ca service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:32.937045 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2503bcab-6ad9-4398-a4ac-7baae4c1f1ca service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:32.937383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2503bcab-6ad9-4398-a4ac-7baae4c1f1ca service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:32.946743 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2503bcab-6ad9-4398-a4ac-7baae4c1f1ca service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:27:32.948406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2503bcab-6ad9-4398-a4ac-7baae4c1f1ca service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:27:32.961044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 753/1513] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:27:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:27:33.843451 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669fbdf9-5c1e-405a-a083-fc535ca4ce85 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:33.844340 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669fbdf9-5c1e-405a-a083-fc535ca4ce85 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:33.844340 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669fbdf9-5c1e-405a-a083-fc535ca4ce85 None None] PATH_INFO: `/v3` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:33.845623 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 761/1514] 10.222.0.17 () {54 vars in 925 bytes} [Tue Oct 4 14:27:33 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:27:33.852462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1510089d-6ed1-4153-a6ae-638ab9fc100d None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:33.853035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1510089d-6ed1-4153-a6ae-638ab9fc100d None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:33.853135 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1510089d-6ed1-4153-a6ae-638ab9fc100d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:33.924035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-1510089d-6ed1-4153-a6ae-638ab9fc100d None None] MFA Rules not processed for user `436deb3d60c54a799700986519b2ef68`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:27:33.925295 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-1510089d-6ed1-4153-a6ae-638ab9fc100d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:27:34.013888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 754/1515] 10.222.0.17 () {58 vars in 1015 bytes} [Tue Oct 4 14:27:33 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:27:34.131397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a79a2f9-80dd-4121-a7b2-0187149b7487 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:34.131672 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a79a2f9-80dd-4121-a7b2-0187149b7487 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:34.131953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a79a2f9-80dd-4121-a7b2-0187149b7487 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:34.133380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 762/1516] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 14:27:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:27:34.140769 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512a7453-4a65-4e58-9776-bed9015a9588 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:34.141060 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512a7453-4a65-4e58-9776-bed9015a9588 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:34.141343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512a7453-4a65-4e58-9776-bed9015a9588 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:34.162151 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-512a7453-4a65-4e58-9776-bed9015a9588 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:27:34.163376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-512a7453-4a65-4e58-9776-bed9015a9588 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:27:34.172048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 755/1517] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 14:27:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:27:34.331528 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a79a2f9-80dd-4121-a7b2-0187149b7487 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:27:34.346636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a79a2f9-80dd-4121-a7b2-0187149b7487 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:27:34.352372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69299b95-ca66-475e-8f34-c030246306aa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:27:34.352989 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69299b95-ca66-475e-8f34-c030246306aa service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:34.353264 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69299b95-ca66-475e-8f34-c030246306aa service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:34.353531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69299b95-ca66-475e-8f34-c030246306aa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:34.364168 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69299b95-ca66-475e-8f34-c030246306aa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:27:34.365804 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69299b95-ca66-475e-8f34-c030246306aa service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:27:34.377344 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 763/1518] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:27:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:27:42.569740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be526f85-8b53-4242-990c-9e172bef4753 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:42.570591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be526f85-8b53-4242-990c-9e172bef4753 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:42.570591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be526f85-8b53-4242-990c-9e172bef4753 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:42.571757 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 756/1519] 10.222.0.64 () {54 vars in 961 bytes} [Tue Oct 4 14:27:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:27:42.578477 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd92316-3455-4bff-b375-43e210129517 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:42.579118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd92316-3455-4bff-b375-43e210129517 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:42.579485 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd92316-3455-4bff-b375-43e210129517 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:42.607332 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-bbd92316-3455-4bff-b375-43e210129517 None None] MFA Rules not processed for user `436deb3d60c54a799700986519b2ef68`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:27:42.608631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-bbd92316-3455-4bff-b375-43e210129517 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:27:42.617480 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 764/1520] 10.222.0.64 () {58 vars in 1058 bytes} [Tue Oct 4 14:27:42 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:27:42.627252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be526f85-8b53-4242-990c-9e172bef4753 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:27:42.642401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be526f85-8b53-4242-990c-9e172bef4753 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:27:42.647681 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f17a14-7638-43f3-bf5b-4167fad1f3ea service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '436deb3d60c54a799700986519b2ef68', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:27:42.648336 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f17a14-7638-43f3-bf5b-4167fad1f3ea service placement] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:42.648610 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f17a14-7638-43f3-bf5b-4167fad1f3ea service placement] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:42.649001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f17a14-7638-43f3-bf5b-4167fad1f3ea service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:42.658571 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f17a14-7638-43f3-bf5b-4167fad1f3ea service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:27:42.660187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f17a14-7638-43f3-bf5b-4167fad1f3ea service placement] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:27:42.672608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 757/1521] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:27:42 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:27:46.141180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8477cf-e0c6-48a3-9816-6bdff259a537 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:46.142102 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8477cf-e0c6-48a3-9816-6bdff259a537 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:46.142102 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8477cf-e0c6-48a3-9816-6bdff259a537 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:46.143734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 765/1522] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 14:27:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:27:46.156163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0764f1-3f08-44c5-bcc9-037faabcc398 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:46.156798 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0764f1-3f08-44c5-bcc9-037faabcc398 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:46.157104 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0764f1-3f08-44c5-bcc9-037faabcc398 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:46.169999 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:27:46.187865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-6c0764f1-3f08-44c5-bcc9-037faabcc398 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:27:46.189171 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-6c0764f1-3f08-44c5-bcc9-037faabcc398 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:27:46.200171 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 758/1523] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 14:27:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:27:46.383759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8477cf-e0c6-48a3-9816-6bdff259a537 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:27:46.403473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8477cf-e0c6-48a3-9816-6bdff259a537 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:27:46.411121 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0dd805-91f8-4177-b71a-b8ae59773332 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:27:46.411794 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0dd805-91f8-4177-b71a-b8ae59773332 service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:46.412104 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0dd805-91f8-4177-b71a-b8ae59773332 service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:46.413239 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0dd805-91f8-4177-b71a-b8ae59773332 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:46.425135 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0dd805-91f8-4177-b71a-b8ae59773332 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:27:46.427044 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0dd805-91f8-4177-b71a-b8ae59773332 service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:27:46.442921 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 766/1524] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:27:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:27:47.696512 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9303e97c-cbcf-478b-841c-3d5f3e80b6cb None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:47.697137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9303e97c-cbcf-478b-841c-3d5f3e80b6cb None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:47.697466 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9303e97c-cbcf-478b-841c-3d5f3e80b6cb None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:47.700206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 759/1525] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 14:27:47 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:27:47.708227 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df7afa5-fc82-497d-86b5-297de7ca2ca7 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:47.708859 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df7afa5-fc82-497d-86b5-297de7ca2ca7 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:47.708859 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df7afa5-fc82-497d-86b5-297de7ca2ca7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:47.732785 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-5df7afa5-fc82-497d-86b5-297de7ca2ca7 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:27:47.734040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5df7afa5-fc82-497d-86b5-297de7ca2ca7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:27:47.742220 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 767/1526] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 14:27:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:27:47.909789 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9303e97c-cbcf-478b-841c-3d5f3e80b6cb None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:27:47.925397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9303e97c-cbcf-478b-841c-3d5f3e80b6cb None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:27:47.931032 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58922b0-276b-489d-9397-8a055af620b2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:27:47.931654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58922b0-276b-489d-9397-8a055af620b2 service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:47.931934 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58922b0-276b-489d-9397-8a055af620b2 service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:47.932209 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58922b0-276b-489d-9397-8a055af620b2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:47.941862 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58922b0-276b-489d-9397-8a055af620b2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:27:47.943465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58922b0-276b-489d-9397-8a055af620b2 service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:27:47.954363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 760/1527] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:27:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:27:52.147610 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f452ba65-5433-49ad-a789-3ebcd1669048 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:52.148176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f452ba65-5433-49ad-a789-3ebcd1669048 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:52.148176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f452ba65-5433-49ad-a789-3ebcd1669048 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:52.149776 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 768/1528] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 14:27:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:27:52.156212 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79296e50-66ea-4cf7-b84f-9c339bd7ab1a None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:52.156835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79296e50-66ea-4cf7-b84f-9c339bd7ab1a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:52.156835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79296e50-66ea-4cf7-b84f-9c339bd7ab1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:52.178325 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-79296e50-66ea-4cf7-b84f-9c339bd7ab1a None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:27:52.179553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-79296e50-66ea-4cf7-b84f-9c339bd7ab1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:27:52.189027 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 761/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 14:27:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:27:52.488150 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f452ba65-5433-49ad-a789-3ebcd1669048 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:27:52.489885 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f452ba65-5433-49ad-a789-3ebcd1669048 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:27:52.553609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f452ba65-5433-49ad-a789-3ebcd1669048 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:27:52.557163 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d8e205-aa7b-475c-b60d-5545621f8106 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:27:52.557764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d8e205-aa7b-475c-b60d-5545621f8106 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:52.558582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d8e205-aa7b-475c-b60d-5545621f8106 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:52.558859 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d8e205-aa7b-475c-b60d-5545621f8106 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:52.568949 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d8e205-aa7b-475c-b60d-5545621f8106 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:27:52.570450 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d8e205-aa7b-475c-b60d-5545621f8106 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:27:52.580704 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 769/1530] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:27:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:27:54.831918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e9a807-1775-45d1-a05c-4c2b4bd8f11e None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:54.832492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e9a807-1775-45d1-a05c-4c2b4bd8f11e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:54.832492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e9a807-1775-45d1-a05c-4c2b4bd8f11e None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:54.833741 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 762/1531] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 14:27:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:27:54.839310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9196aa0-8dd8-44da-90e6-be3d71465877 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:54.839872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9196aa0-8dd8-44da-90e6-be3d71465877 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:54.840018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9196aa0-8dd8-44da-90e6-be3d71465877 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:54.847687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=72 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:27:54.868601 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-a9196aa0-8dd8-44da-90e6-be3d71465877 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:27:54.871525 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a9196aa0-8dd8-44da-90e6-be3d71465877 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:27:54.885130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 770/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 14:27:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:27:55.186923 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e9a807-1775-45d1-a05c-4c2b4bd8f11e None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:27:55.199413 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e9a807-1775-45d1-a05c-4c2b4bd8f11e None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:27:55.203065 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69abd20f-ebb7-41bd-b3c3-7bd329588a84 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:27:55.203674 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69abd20f-ebb7-41bd-b3c3-7bd329588a84 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:55.203961 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69abd20f-ebb7-41bd-b3c3-7bd329588a84 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:55.204236 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69abd20f-ebb7-41bd-b3c3-7bd329588a84 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:55.214671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69abd20f-ebb7-41bd-b3c3-7bd329588a84 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:27:55.216276 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69abd20f-ebb7-41bd-b3c3-7bd329588a84 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:27:55.226522 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 763/1533] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:27:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:27:59.123569 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8bcb3e-04e4-4f5c-98ca-d6cab33a07ad None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:59.124168 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8bcb3e-04e4-4f5c-98ca-d6cab33a07ad None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:59.124168 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8bcb3e-04e4-4f5c-98ca-d6cab33a07ad None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:59.125715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 771/1534] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 14:27:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:27:59.132844 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245459da-f3d0-45ef-94bc-b051ab94c9a6 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:59.133437 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245459da-f3d0-45ef-94bc-b051ab94c9a6 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:59.133759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245459da-f3d0-45ef-94bc-b051ab94c9a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:59.160819 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-245459da-f3d0-45ef-94bc-b051ab94c9a6 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:27:59.162147 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-245459da-f3d0-45ef-94bc-b051ab94c9a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:27:59.172734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 764/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 14:27:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:27:59.508786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8bcb3e-04e4-4f5c-98ca-d6cab33a07ad None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:27:59.510601 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-cf8bcb3e-04e4-4f5c-98ca-d6cab33a07ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:27:59.574511 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8bcb3e-04e4-4f5c-98ca-d6cab33a07ad None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:27:59.578580 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8f2432-503a-4dfb-ad87-2a7fe30a2dfe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:27:59.579197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8f2432-503a-4dfb-ad87-2a7fe30a2dfe service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:27:59.579471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8f2432-503a-4dfb-ad87-2a7fe30a2dfe service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:27:59.579736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8f2432-503a-4dfb-ad87-2a7fe30a2dfe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:27:59.590014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8f2432-503a-4dfb-ad87-2a7fe30a2dfe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:27:59.591462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8f2432-503a-4dfb-ad87-2a7fe30a2dfe service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:27:59.602531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 772/1536] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:27:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:28:02.007936 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbd7ec9-e8d7-4287-8643-f9698e3aa71a None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:28:02.008756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbd7ec9-e8d7-4287-8643-f9698e3aa71a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:28:02.008756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbd7ec9-e8d7-4287-8643-f9698e3aa71a None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:28:02.010254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 765/1537] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 14:28:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:28:02.017638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9281cdbd-586a-4ac1-ac97-f65e04e60e03 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:28:02.018260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9281cdbd-586a-4ac1-ac97-f65e04e60e03 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:28:02.018260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9281cdbd-586a-4ac1-ac97-f65e04e60e03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:28:02.041067 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-9281cdbd-586a-4ac1-ac97-f65e04e60e03 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:28:02.042382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9281cdbd-586a-4ac1-ac97-f65e04e60e03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:28:02.051715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 773/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 14:28:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:28:02.377653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbd7ec9-e8d7-4287-8643-f9698e3aa71a None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:28:02.379449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-dfbd7ec9-e8d7-4287-8643-f9698e3aa71a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:28:02.393233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbd7ec9-e8d7-4287-8643-f9698e3aa71a None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:28:02.397267 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358b240f-cd32-48a7-b6ca-01226d47988f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:28:02.397887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358b240f-cd32-48a7-b6ca-01226d47988f service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:28:02.398163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358b240f-cd32-48a7-b6ca-01226d47988f service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:28:02.398431 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358b240f-cd32-48a7-b6ca-01226d47988f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:28:02.408395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358b240f-cd32-48a7-b6ca-01226d47988f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:28:02.409834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358b240f-cd32-48a7-b6ca-01226d47988f service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:28:02.420461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 766/1539] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:28:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:28:12.034349 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=31 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:28:12.415438 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=26 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:28.520696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685c5b9d-1fdf-49ae-bcf6-5dc000f9ef5b None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:28.521326 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685c5b9d-1fdf-49ae-bcf6-5dc000f9ef5b None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:28.521326 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685c5b9d-1fdf-49ae-bcf6-5dc000f9ef5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:28.526193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060fd2d7-afb7-4e85-a8f2-8cd85e324407 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:28.526762 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060fd2d7-afb7-4e85-a8f2-8cd85e324407 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:28.526762 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060fd2d7-afb7-4e85-a8f2-8cd85e324407 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:28.547687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-685c5b9d-1fdf-49ae-bcf6-5dc000f9ef5b None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:28.548916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-685c5b9d-1fdf-49ae-bcf6-5dc000f9ef5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:28.557500 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-060fd2d7-afb7-4e85-a8f2-8cd85e324407 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:28.558754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-060fd2d7-afb7-4e85-a8f2-8cd85e324407 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:28.562131 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 774/1540] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:28.572512 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05f6da0-22a4-4487-ae54-498490367832 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:28.572775 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05f6da0-22a4-4487-ae54-498490367832 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:28.573057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05f6da0-22a4-4487-ae54-498490367832 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:28.578146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 767/1541] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:28.588078 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eefd1f6-f4e3-4ffa-8499-be2b080b7156 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:28.588445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eefd1f6-f4e3-4ffa-8499-be2b080b7156 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:28.588671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eefd1f6-f4e3-4ffa-8499-be2b080b7156 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:28.609692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-c05f6da0-22a4-4487-ae54-498490367832 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:28.611072 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c05f6da0-22a4-4487-ae54-498490367832 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:28.619720 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 775/1542] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:29:28.625496 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfdbe79-c6b4-43b9-ae73-cd362a358e1c None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:28.625708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfdbe79-c6b4-43b9-ae73-cd362a358e1c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:28.625960 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfdbe79-c6b4-43b9-ae73-cd362a358e1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:28.634712 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-3eefd1f6-f4e3-4ffa-8499-be2b080b7156 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:28.636404 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-3eefd1f6-f4e3-4ffa-8499-be2b080b7156 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:28.656095 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-6dfdbe79-c6b4-43b9-ae73-cd362a358e1c None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:28.657397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6dfdbe79-c6b4-43b9-ae73-cd362a358e1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:28.666221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 768/1543] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:29:28.667190 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eefd1f6-f4e3-4ffa-8499-be2b080b7156 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:28.675655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 776/1544] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:28.676577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dfdbe79-c6b4-43b9-ae73-cd362a358e1c None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:28.686848 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eefd1f6-f4e3-4ffa-8499-be2b080b7156 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:28.687462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dfdbe79-c6b4-43b9-ae73-cd362a358e1c None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:28.705074 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8452b12-18b7-4804-bada-1f46ab479252 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:28.705713 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8452b12-18b7-4804-bada-1f46ab479252 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:28.706011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8452b12-18b7-4804-bada-1f46ab479252 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:28.706288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8452b12-18b7-4804-bada-1f46ab479252 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:28.707096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8452b12-18b7-4804-bada-1f46ab479252 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:29:28.707681 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8452b12-18b7-4804-bada-1f46ab479252 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:28.709356 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8452b12-18b7-4804-bada-1f46ab479252 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:28.713069 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9d7a68-ef6c-47c3-8499-e162998178fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:28.713712 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9d7a68-ef6c-47c3-8499-e162998178fa admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:28.713996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9d7a68-ef6c-47c3-8499-e162998178fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:28.714272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9d7a68-ef6c-47c3-8499-e162998178fa admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:28.714934 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9d7a68-ef6c-47c3-8499-e162998178fa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:29:28.715241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9d7a68-ef6c-47c3-8499-e162998178fa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:28.716274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: WARNING py.warnings [None req-eb9d7a68-ef6c-47c3-8499-e162998178fa admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 04 14:29:28.716274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: warnings.warn(msg) Oct 04 14:29:28.716274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: Oct 04 14:29:28.717196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9d7a68-ef6c-47c3-8499-e162998178fa admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:28.739118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 777/1545] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:29:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:29:28.740471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebe0c88-8f5d-4894-9097-8c59c6436ac3 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:28.740759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebe0c88-8f5d-4894-9097-8c59c6436ac3 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:28.741032 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebe0c88-8f5d-4894-9097-8c59c6436ac3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:28.741682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 769/1546] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:29:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:29:28.743908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8e4e44-df8b-4772-a81c-376c89117712 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:28.744207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8e4e44-df8b-4772-a81c-376c89117712 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:28.744506 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8e4e44-df8b-4772-a81c-376c89117712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:28.773314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-ec8e4e44-df8b-4772-a81c-376c89117712 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:28.774888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ec8e4e44-df8b-4772-a81c-376c89117712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:28.777311 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-9ebe0c88-8f5d-4894-9097-8c59c6436ac3 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:28.778582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9ebe0c88-8f5d-4894-9097-8c59c6436ac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:28.783741 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 770/1547] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:28.785032 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585337b9-de51-4b22-9906-1244cc688373 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:28.785351 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585337b9-de51-4b22-9906-1244cc688373 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:28.785648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585337b9-de51-4b22-9906-1244cc688373 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:28.787308 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 778/1548] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:28.788248 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebe0c88-8f5d-4894-9097-8c59c6436ac3 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:28.799408 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebe0c88-8f5d-4894-9097-8c59c6436ac3 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:28.804334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da732538-7e17-41f3-bf4a-c66d2167bd5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:28.805733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da732538-7e17-41f3-bf4a-c66d2167bd5b admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:28.806088 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da732538-7e17-41f3-bf4a-c66d2167bd5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:28.806528 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da732538-7e17-41f3-bf4a-c66d2167bd5b admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:28.807074 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da732538-7e17-41f3-bf4a-c66d2167bd5b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:28.808284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: WARNING py.warnings [None req-da732538-7e17-41f3-bf4a-c66d2167bd5b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 14:29:28.808284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: warnings.warn(msg) Oct 04 14:29:28.808284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: Oct 04 14:29:28.809893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da732538-7e17-41f3-bf4a-c66d2167bd5b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:28.824876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-585337b9-de51-4b22-9906-1244cc688373 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:28.827379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-585337b9-de51-4b22-9906-1244cc688373 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:28.842631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 771/1549] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:28.844204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585337b9-de51-4b22-9906-1244cc688373 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:28.847782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 779/1550] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:29:28 2022] POST /identity/v3/projects => generated 393 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:28.849816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112306a8-d7c2-4c8e-ad5c-fe64f064e85c None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:28.850487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112306a8-d7c2-4c8e-ad5c-fe64f064e85c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:28.851006 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112306a8-d7c2-4c8e-ad5c-fe64f064e85c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:28.859797 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585337b9-de51-4b22-9906-1244cc688373 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:28.876058 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-112306a8-d7c2-4c8e-ad5c-fe64f064e85c None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:28.877416 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-112306a8-d7c2-4c8e-ad5c-fe64f064e85c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:28.932243 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e572b15-0c3e-4fb6-b20f-3049674f6829 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:28.932904 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e572b15-0c3e-4fb6-b20f-3049674f6829 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:28.933188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e572b15-0c3e-4fb6-b20f-3049674f6829 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:28.933462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e572b15-0c3e-4fb6-b20f-3049674f6829 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:28.934115 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e572b15-0c3e-4fb6-b20f-3049674f6829 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:28.936073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e572b15-0c3e-4fb6-b20f-3049674f6829 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:28.954097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 780/1551] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:28.955381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112306a8-d7c2-4c8e-ad5c-fe64f064e85c None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:28.969375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112306a8-d7c2-4c8e-ad5c-fe64f064e85c None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:28.975187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 772/1552] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:29:28 2022] POST /identity/v3/projects => generated 383 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:28.975187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79148d8b-6ed9-49ee-a48a-81f9220cb38d None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:28.975187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79148d8b-6ed9-49ee-a48a-81f9220cb38d None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:28.975187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79148d8b-6ed9-49ee-a48a-81f9220cb38d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:28.996288 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-79148d8b-6ed9-49ee-a48a-81f9220cb38d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:28.997769 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-79148d8b-6ed9-49ee-a48a-81f9220cb38d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:29.034663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980b216c-34b7-4712-a28a-bdc919e4ad1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.035633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980b216c-34b7-4712-a28a-bdc919e4ad1e admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.035633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980b216c-34b7-4712-a28a-bdc919e4ad1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.035868 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980b216c-34b7-4712-a28a-bdc919e4ad1e admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.036489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980b216c-34b7-4712-a28a-bdc919e4ad1e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:29:29.036796 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980b216c-34b7-4712-a28a-bdc919e4ad1e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.038411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980b216c-34b7-4712-a28a-bdc919e4ad1e admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.052240 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 781/1553] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:29:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:29:29.054391 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205114f0-abab-44cb-b45d-cc4af2372ad5 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.055394 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205114f0-abab-44cb-b45d-cc4af2372ad5 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.055873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205114f0-abab-44cb-b45d-cc4af2372ad5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.072242 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 773/1554] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:29.073126 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79148d8b-6ed9-49ee-a48a-81f9220cb38d None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.083253 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79148d8b-6ed9-49ee-a48a-81f9220cb38d None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.086042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-205114f0-abab-44cb-b45d-cc4af2372ad5 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:29.087257 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-205114f0-abab-44cb-b45d-cc4af2372ad5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:29.089055 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624b42ad-4be9-423e-86af-baa7869dee0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.089733 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624b42ad-4be9-423e-86af-baa7869dee0f admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.090077 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624b42ad-4be9-423e-86af-baa7869dee0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.090359 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624b42ad-4be9-423e-86af-baa7869dee0f admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.091064 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624b42ad-4be9-423e-86af-baa7869dee0f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.092520 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: WARNING py.warnings [None req-624b42ad-4be9-423e-86af-baa7869dee0f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 14:29:29.092520 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: warnings.warn(msg) Oct 04 14:29:29.092520 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: Oct 04 14:29:29.093881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624b42ad-4be9-423e-86af-baa7869dee0f admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.095401 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 782/1555] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:29.096663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d47d69-d94c-40de-89ac-dbc17dd13e98 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.096941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d47d69-d94c-40de-89ac-dbc17dd13e98 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.097213 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d47d69-d94c-40de-89ac-dbc17dd13e98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.125322 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-e1d47d69-d94c-40de-89ac-dbc17dd13e98 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:29.126754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 774/1556] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:29:29 2022] POST /identity/v3/users => generated 297 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:29.126926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-e1d47d69-d94c-40de-89ac-dbc17dd13e98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:29.127743 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f946b26-7238-435e-bec1-04ee4ecf8d2b None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.128033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f946b26-7238-435e-bec1-04ee4ecf8d2b None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.128303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f946b26-7238-435e-bec1-04ee4ecf8d2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.139331 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 783/1557] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:29.141220 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184a2fe7-6bbe-4715-ba60-22d840526c66 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.141502 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184a2fe7-6bbe-4715-ba60-22d840526c66 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.141775 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184a2fe7-6bbe-4715-ba60-22d840526c66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.153038 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-9f946b26-7238-435e-bec1-04ee4ecf8d2b None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:29.154156 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9f946b26-7238-435e-bec1-04ee4ecf8d2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:29.165356 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 775/1558] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:29.166132 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-184a2fe7-6bbe-4715-ba60-22d840526c66 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:29.166816 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f946b26-7238-435e-bec1-04ee4ecf8d2b None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.167689 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-184a2fe7-6bbe-4715-ba60-22d840526c66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:29.176094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 784/1559] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:29.177053 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f946b26-7238-435e-bec1-04ee4ecf8d2b None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.181147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184a2fe7-6bbe-4715-ba60-22d840526c66 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.182673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7b3145-d4a1-4752-83eb-c925c517f1d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.183419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7b3145-d4a1-4752-83eb-c925c517f1d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.183602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7b3145-d4a1-4752-83eb-c925c517f1d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.183852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7b3145-d4a1-4752-83eb-c925c517f1d7 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.184558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7b3145-d4a1-4752-83eb-c925c517f1d7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.186442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7b3145-d4a1-4752-83eb-c925c517f1d7 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.191474 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184a2fe7-6bbe-4715-ba60-22d840526c66 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.196756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f285b3fb-2302-49fb-8e01-3c520c4bcb77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.197904 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f285b3fb-2302-49fb-8e01-3c520c4bcb77 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.198226 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f285b3fb-2302-49fb-8e01-3c520c4bcb77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.198515 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f285b3fb-2302-49fb-8e01-3c520c4bcb77 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.199618 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f285b3fb-2302-49fb-8e01-3c520c4bcb77 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.201793 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f285b3fb-2302-49fb-8e01-3c520c4bcb77 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.211664 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 776/1560] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:29:29 2022] POST /identity/v3/users => generated 292 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:29.212956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7b3145-d4a1-4752-83eb-c925c517f1d7 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.217779 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:29.220365 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:29.227071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7b3145-d4a1-4752-83eb-c925c517f1d7 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.232799 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905a2f00-0cb2-41c6-82b4-e667d7dd9c33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.233410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905a2f00-0cb2-41c6-82b4-e667d7dd9c33 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.233682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905a2f00-0cb2-41c6-82b4-e667d7dd9c33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.233950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905a2f00-0cb2-41c6-82b4-e667d7dd9c33 admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.234593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905a2f00-0cb2-41c6-82b4-e667d7dd9c33 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:29:29.234891 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905a2f00-0cb2-41c6-82b4-e667d7dd9c33 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.236584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905a2f00-0cb2-41c6-82b4-e667d7dd9c33 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.239255 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 785/1561] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:29:29 2022] POST /identity/v3/projects => generated 373 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:29.241206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7039cb9-7d04-4626-903b-2181767406ad None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.241361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7039cb9-7d04-4626-903b-2181767406ad None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.241677 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7039cb9-7d04-4626-903b-2181767406ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.250611 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 777/1562] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:29:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:29:29.251569 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905a2f00-0cb2-41c6-82b4-e667d7dd9c33 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.261704 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905a2f00-0cb2-41c6-82b4-e667d7dd9c33 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.270926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-f7039cb9-7d04-4626-903b-2181767406ad None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:29.272402 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f7039cb9-7d04-4626-903b-2181767406ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:29.338067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8813022d-b280-4d52-9893-602a2548f3ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.339159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8813022d-b280-4d52-9893-602a2548f3ad admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.339663 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8813022d-b280-4d52-9893-602a2548f3ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.340157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8813022d-b280-4d52-9893-602a2548f3ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.340946 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8813022d-b280-4d52-9893-602a2548f3ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.341486 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 786/1563] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:29.342594 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8813022d-b280-4d52-9893-602a2548f3ad admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.342833 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7039cb9-7d04-4626-903b-2181767406ad None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.355586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7039cb9-7d04-4626-903b-2181767406ad None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.357829 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 778/1564] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:29 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:29.359620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7e30d2-203c-4d7f-834c-6a124039b7db None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.359912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7e30d2-203c-4d7f-834c-6a124039b7db None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.360194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7e30d2-203c-4d7f-834c-6a124039b7db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.363441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6306918-3d3e-4dad-be32-3f961ca83501 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.364067 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6306918-3d3e-4dad-be32-3f961ca83501 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.364366 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6306918-3d3e-4dad-be32-3f961ca83501 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.364641 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6306918-3d3e-4dad-be32-3f961ca83501 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.365294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6306918-3d3e-4dad-be32-3f961ca83501 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:29:29.365592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6306918-3d3e-4dad-be32-3f961ca83501 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.367940 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6306918-3d3e-4dad-be32-3f961ca83501 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.381533 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 787/1565] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:29:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:29:29.382901 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-ce7e30d2-203c-4d7f-834c-6a124039b7db None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:29.384199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ce7e30d2-203c-4d7f-834c-6a124039b7db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:29.389311 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6306918-3d3e-4dad-be32-3f961ca83501 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.392424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 779/1566] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:29.393235 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7e30d2-203c-4d7f-834c-6a124039b7db None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.400867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6306918-3d3e-4dad-be32-3f961ca83501 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.406425 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7e30d2-203c-4d7f-834c-6a124039b7db None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.407486 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06d9aa0-c58f-4db9-b8c0-ef41ebbb039c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.408559 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06d9aa0-c58f-4db9-b8c0-ef41ebbb039c admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.408883 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06d9aa0-c58f-4db9-b8c0-ef41ebbb039c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.409096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06d9aa0-c58f-4db9-b8c0-ef41ebbb039c admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.409747 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06d9aa0-c58f-4db9-b8c0-ef41ebbb039c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:29:29.410046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06d9aa0-c58f-4db9-b8c0-ef41ebbb039c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.411808 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06d9aa0-c58f-4db9-b8c0-ef41ebbb039c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.413495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a48455c-60b9-41b3-9f6e-9ac06e0d1aea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.414908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a48455c-60b9-41b3-9f6e-9ac06e0d1aea admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.415254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a48455c-60b9-41b3-9f6e-9ac06e0d1aea admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.415602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a48455c-60b9-41b3-9f6e-9ac06e0d1aea admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.416352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a48455c-60b9-41b3-9f6e-9ac06e0d1aea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.418080 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a48455c-60b9-41b3-9f6e-9ac06e0d1aea admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.428174 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 788/1567] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:29:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:29:29.433155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 780/1568] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:29 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:29.433654 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06d9aa0-c58f-4db9-b8c0-ef41ebbb039c admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.434254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a48455c-60b9-41b3-9f6e-9ac06e0d1aea admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.445814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a48455c-60b9-41b3-9f6e-9ac06e0d1aea admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.446579 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06d9aa0-c58f-4db9-b8c0-ef41ebbb039c admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.451445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bd4e90-1e74-4923-a0a4-9e0901c655bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.451999 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bd4e90-1e74-4923-a0a4-9e0901c655bc admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.452226 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bd4e90-1e74-4923-a0a4-9e0901c655bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.452434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1441c3ab-a88d-4eb6-b923-5ee48473fb34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.452612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bd4e90-1e74-4923-a0a4-9e0901c655bc admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.453154 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09bd4e90-1e74-4923-a0a4-9e0901c655bc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.453300 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1441c3ab-a88d-4eb6-b923-5ee48473fb34 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.453687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1441c3ab-a88d-4eb6-b923-5ee48473fb34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.454109 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1441c3ab-a88d-4eb6-b923-5ee48473fb34 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.454888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1441c3ab-a88d-4eb6-b923-5ee48473fb34 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.455072 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09bd4e90-1e74-4923-a0a4-9e0901c655bc admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.457695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1441c3ab-a88d-4eb6-b923-5ee48473fb34 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.478745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 789/1569] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:29:29 2022] POST /identity/v3/users => generated 287 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:29.479523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bd4e90-1e74-4923-a0a4-9e0901c655bc admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.489732 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bd4e90-1e74-4923-a0a4-9e0901c655bc admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.494505 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf322ac-542e-4fdc-9df8-ebfece05f9ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.495008 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 781/1570] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:29:29 2022] POST /identity/v3/projects => generated 385 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:29.495114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf322ac-542e-4fdc-9df8-ebfece05f9ca admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.495364 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf322ac-542e-4fdc-9df8-ebfece05f9ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.495680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf322ac-542e-4fdc-9df8-ebfece05f9ca admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.495870 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1441c3ab-a88d-4eb6-b923-5ee48473fb34 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.496351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf322ac-542e-4fdc-9df8-ebfece05f9ca admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:29:29.496649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf322ac-542e-4fdc-9df8-ebfece05f9ca admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.498480 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf322ac-542e-4fdc-9df8-ebfece05f9ca admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.511733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 790/1571] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:29:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:29:29.517683 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf322ac-542e-4fdc-9df8-ebfece05f9ca admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.522230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf322ac-542e-4fdc-9df8-ebfece05f9ca admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.583355 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2338da-e81f-4608-a16a-44a16174ed64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.584436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2338da-e81f-4608-a16a-44a16174ed64 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.584898 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2338da-e81f-4608-a16a-44a16174ed64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.585388 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2338da-e81f-4608-a16a-44a16174ed64 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.586971 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2338da-e81f-4608-a16a-44a16174ed64 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.590459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2338da-e81f-4608-a16a-44a16174ed64 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.596092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1441c3ab-a88d-4eb6-b923-5ee48473fb34 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.604379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3525b8e0-96e8-4f14-a6fa-5f2d985924a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.605354 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3525b8e0-96e8-4f14-a6fa-5f2d985924a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.605810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3525b8e0-96e8-4f14-a6fa-5f2d985924a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.606260 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3525b8e0-96e8-4f14-a6fa-5f2d985924a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.607253 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3525b8e0-96e8-4f14-a6fa-5f2d985924a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.610052 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3525b8e0-96e8-4f14-a6fa-5f2d985924a7 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.622184 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 782/1572] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:29 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:29.623208 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3525b8e0-96e8-4f14-a6fa-5f2d985924a7 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.634445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 791/1573] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:29:29 2022] POST /identity/v3/projects => generated 369 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:29.635381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2338da-e81f-4608-a16a-44a16174ed64 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.636492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3525b8e0-96e8-4f14-a6fa-5f2d985924a7 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.662527 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:29.698776 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:29.715558 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2338da-e81f-4608-a16a-44a16174ed64 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.722700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c003e53-2ad9-4c07-af6f-cd7465768980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.723068 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb07aac-49c5-4b56-8bbe-792c75a82a18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.723605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c003e53-2ad9-4c07-af6f-cd7465768980 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.724042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c003e53-2ad9-4c07-af6f-cd7465768980 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.724282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb07aac-49c5-4b56-8bbe-792c75a82a18 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.724508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb07aac-49c5-4b56-8bbe-792c75a82a18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.724629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c003e53-2ad9-4c07-af6f-cd7465768980 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.724940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb07aac-49c5-4b56-8bbe-792c75a82a18 admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.725573 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c003e53-2ad9-4c07-af6f-cd7465768980 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.725991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb07aac-49c5-4b56-8bbe-792c75a82a18 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:29:29.727081 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb07aac-49c5-4b56-8bbe-792c75a82a18 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.729061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c003e53-2ad9-4c07-af6f-cd7465768980 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.729793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb07aac-49c5-4b56-8bbe-792c75a82a18 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.744351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 792/1574] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:29 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:29.745024 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c003e53-2ad9-4c07-af6f-cd7465768980 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.754346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 783/1575] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:29:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:29:29.754346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb07aac-49c5-4b56-8bbe-792c75a82a18 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.755230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c003e53-2ad9-4c07-af6f-cd7465768980 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.760050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a013996-7218-43a0-a73f-201d4ae190ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.760687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a013996-7218-43a0-a73f-201d4ae190ac admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.760963 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a013996-7218-43a0-a73f-201d4ae190ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.761230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a013996-7218-43a0-a73f-201d4ae190ac admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.761860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a013996-7218-43a0-a73f-201d4ae190ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.763072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb07aac-49c5-4b56-8bbe-792c75a82a18 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.763866 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a013996-7218-43a0-a73f-201d4ae190ac admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.771156 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e314e70a-17dd-4c58-ba13-97a6a2260008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.771916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e314e70a-17dd-4c58-ba13-97a6a2260008 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.772692 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e314e70a-17dd-4c58-ba13-97a6a2260008 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.772692 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e314e70a-17dd-4c58-ba13-97a6a2260008 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.774583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e314e70a-17dd-4c58-ba13-97a6a2260008 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.775698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e314e70a-17dd-4c58-ba13-97a6a2260008 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.787554 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 784/1576] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:29 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:29.788915 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e314e70a-17dd-4c58-ba13-97a6a2260008 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.795419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 793/1577] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:29:29 2022] POST /identity/v3/projects => generated 389 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:29.796428 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a013996-7218-43a0-a73f-201d4ae190ac admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.802117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e314e70a-17dd-4c58-ba13-97a6a2260008 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.863215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a013996-7218-43a0-a73f-201d4ae190ac admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.868003 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4194b5d-9422-41f4-9aab-c07c0509987a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.868633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4194b5d-9422-41f4-9aab-c07c0509987a admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.868767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f18e614-8cdc-4ea0-b4e3-a41cf6b29c3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.868895 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4194b5d-9422-41f4-9aab-c07c0509987a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.869167 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4194b5d-9422-41f4-9aab-c07c0509987a admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.869319 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f18e614-8cdc-4ea0-b4e3-a41cf6b29c3e admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.869596 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f18e614-8cdc-4ea0-b4e3-a41cf6b29c3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.869797 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4194b5d-9422-41f4-9aab-c07c0509987a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.869920 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f18e614-8cdc-4ea0-b4e3-a41cf6b29c3e admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.870487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f18e614-8cdc-4ea0-b4e3-a41cf6b29c3e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.871886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4194b5d-9422-41f4-9aab-c07c0509987a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.872431 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f18e614-8cdc-4ea0-b4e3-a41cf6b29c3e admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.896143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 785/1578] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:29:29 2022] POST /identity/v3/users => generated 293 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:29.897265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f18e614-8cdc-4ea0-b4e3-a41cf6b29c3e admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.902949 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 794/1579] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:29:29 2022] POST /identity/v3/projects => generated 375 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:29.904015 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4194b5d-9422-41f4-9aab-c07c0509987a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:29.909299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f18e614-8cdc-4ea0-b4e3-a41cf6b29c3e admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.975409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809de7f8-e77a-4e77-bf08-8c9dcaf8fc0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.976191 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809de7f8-e77a-4e77-bf08-8c9dcaf8fc0d admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.976692 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809de7f8-e77a-4e77-bf08-8c9dcaf8fc0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.976980 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809de7f8-e77a-4e77-bf08-8c9dcaf8fc0d admin admin] PATH_INFO: `/v3/projects/ec24f1b720bd42ebb37a36d65757aacd/users/2b4c980bac3e444faa0e6f1a1d270cfa/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.984715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4194b5d-9422-41f4-9aab-c07c0509987a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:29.989912 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73008b95-aff6-4e21-a520-881e84130fe7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:29.990527 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73008b95-aff6-4e21-a520-881e84130fe7 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:29.990805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73008b95-aff6-4e21-a520-881e84130fe7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:29.991221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73008b95-aff6-4e21-a520-881e84130fe7 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:29.991723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73008b95-aff6-4e21-a520-881e84130fe7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:29.993956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73008b95-aff6-4e21-a520-881e84130fe7 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:29.997704 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809de7f8-e77a-4e77-bf08-8c9dcaf8fc0d admin admin] RBAC: Authorizing `identity:create_grant(project_id=ec24f1b720bd42ebb37a36d65757aacd, user_id=2b4c980bac3e444faa0e6f1a1d270cfa, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.001747 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809de7f8-e77a-4e77-bf08-8c9dcaf8fc0d admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.018159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 795/1580] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:29:29 2022] POST /identity/v3/users => generated 285 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:30.023226 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73008b95-aff6-4e21-a520-881e84130fe7 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.023991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 786/1581] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:29 2022] PUT /identity/v3/projects/ec24f1b720bd42ebb37a36d65757aacd/users/2b4c980bac3e444faa0e6f1a1d270cfa/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:30.023991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809de7f8-e77a-4e77-bf08-8c9dcaf8fc0d admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.036400 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73008b95-aff6-4e21-a520-881e84130fe7 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.046944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:30.080157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:30.106590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809de7f8-e77a-4e77-bf08-8c9dcaf8fc0d admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.108522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239a4233-b17e-44d1-b001-db56ba132703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.109335 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239a4233-b17e-44d1-b001-db56ba132703 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.109613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239a4233-b17e-44d1-b001-db56ba132703 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.109886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239a4233-b17e-44d1-b001-db56ba132703 admin admin] PATH_INFO: `/v3/projects/0f752b93233147d59c19b3dc7f5b42c4/users/a99a904626e44182a020368a63c4dad3/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.111572 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0d9bc6-1460-4771-8740-54868f228e20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.112185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0d9bc6-1460-4771-8740-54868f228e20 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.112572 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0d9bc6-1460-4771-8740-54868f228e20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.112846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0d9bc6-1460-4771-8740-54868f228e20 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.113413 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0d9bc6-1460-4771-8740-54868f228e20 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.115342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0d9bc6-1460-4771-8740-54868f228e20 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.127653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239a4233-b17e-44d1-b001-db56ba132703 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f752b93233147d59c19b3dc7f5b42c4, user_id=a99a904626e44182a020368a63c4dad3, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.128746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: WARNING py.warnings [None req-239a4233-b17e-44d1-b001-db56ba132703 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 14:29:30.128746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: warnings.warn(msg) Oct 04 14:29:30.128746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: Oct 04 14:29:30.130755 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239a4233-b17e-44d1-b001-db56ba132703 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.150147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 796/1582] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:30 2022] PUT /identity/v3/projects/0f752b93233147d59c19b3dc7f5b42c4/users/a99a904626e44182a020368a63c4dad3/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:30.151140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239a4233-b17e-44d1-b001-db56ba132703 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.162907 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 787/1583] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:29:30 2022] POST /identity/v3/projects => generated 379 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:30.162907 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0d9bc6-1460-4771-8740-54868f228e20 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.235383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0d9bc6-1460-4771-8740-54868f228e20 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.240261 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bc35c6-ddb9-4372-b232-46044272bb1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.240903 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bc35c6-ddb9-4372-b232-46044272bb1a admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.241187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bc35c6-ddb9-4372-b232-46044272bb1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.241466 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bc35c6-ddb9-4372-b232-46044272bb1a admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.242663 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bc35c6-ddb9-4372-b232-46044272bb1a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.244908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bc35c6-ddb9-4372-b232-46044272bb1a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.252586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239a4233-b17e-44d1-b001-db56ba132703 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.262047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7daea3a0-bc49-4541-8265-e21df2e10b53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.263219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7daea3a0-bc49-4541-8265-e21df2e10b53 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.263739 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7daea3a0-bc49-4541-8265-e21df2e10b53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.264209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7daea3a0-bc49-4541-8265-e21df2e10b53 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.265422 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7daea3a0-bc49-4541-8265-e21df2e10b53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.268577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7daea3a0-bc49-4541-8265-e21df2e10b53 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.268716 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 788/1584] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:29:30 2022] POST /identity/v3/users => generated 295 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:30.269417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bc35c6-ddb9-4372-b232-46044272bb1a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.283312 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bc35c6-ddb9-4372-b232-46044272bb1a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.284513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 797/1585] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:30 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:30.285527 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7daea3a0-bc49-4541-8265-e21df2e10b53 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.288383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faad752-5a28-4f03-a538-a9ab38f6875e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.289639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faad752-5a28-4f03-a538-a9ab38f6875e admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.290173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faad752-5a28-4f03-a538-a9ab38f6875e admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.290599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faad752-5a28-4f03-a538-a9ab38f6875e admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.291301 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faad752-5a28-4f03-a538-a9ab38f6875e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.292942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faad752-5a28-4f03-a538-a9ab38f6875e admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.296817 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7daea3a0-bc49-4541-8265-e21df2e10b53 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.301987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b12c72-d0ee-4750-aa8d-f47aa300cf65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.302598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b12c72-d0ee-4750-aa8d-f47aa300cf65 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.302874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b12c72-d0ee-4750-aa8d-f47aa300cf65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.303159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b12c72-d0ee-4750-aa8d-f47aa300cf65 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.303777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b12c72-d0ee-4750-aa8d-f47aa300cf65 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.305419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 789/1586] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:30 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:30.306122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b12c72-d0ee-4750-aa8d-f47aa300cf65 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.306273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faad752-5a28-4f03-a538-a9ab38f6875e admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.318205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faad752-5a28-4f03-a538-a9ab38f6875e admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.324197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10c2f65-96c5-463c-83c0-79161714e5cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.324846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10c2f65-96c5-463c-83c0-79161714e5cc admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.325128 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10c2f65-96c5-463c-83c0-79161714e5cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.325404 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10c2f65-96c5-463c-83c0-79161714e5cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.326061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10c2f65-96c5-463c-83c0-79161714e5cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.327711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10c2f65-96c5-463c-83c0-79161714e5cc admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.338020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 798/1587] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:29:30 2022] POST /identity/v3/users => generated 288 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:30.339450 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f479c2-1f3f-4bba-b3ee-20662de608ee None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.339728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f479c2-1f3f-4bba-b3ee-20662de608ee None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.340018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f479c2-1f3f-4bba-b3ee-20662de608ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.343540 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 790/1588] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:30 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:30.345370 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cd96a0-0cd3-49c0-bba4-c54e503f054a None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.345781 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cd96a0-0cd3-49c0-bba4-c54e503f054a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.346200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cd96a0-0cd3-49c0-bba4-c54e503f054a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.362660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-a2f479c2-1f3f-4bba-b3ee-20662de608ee None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:30.364189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a2f479c2-1f3f-4bba-b3ee-20662de608ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:30.374692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 799/1589] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:30.378019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f479c2-1f3f-4bba-b3ee-20662de608ee None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.381374 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-b6cd96a0-0cd3-49c0-bba4-c54e503f054a None None] MFA Rules not processed for user `a99a904626e44182a020368a63c4dad3`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:30.383001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-b6cd96a0-0cd3-49c0-bba4-c54e503f054a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:30.389342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f479c2-1f3f-4bba-b3ee-20662de608ee None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.394392 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c95ff0-c706-4a95-827e-f14bbb9dfa3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.395161 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c95ff0-c706-4a95-827e-f14bbb9dfa3a admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.395440 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c95ff0-c706-4a95-827e-f14bbb9dfa3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.395711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c95ff0-c706-4a95-827e-f14bbb9dfa3a admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.396378 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c95ff0-c706-4a95-827e-f14bbb9dfa3a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.398851 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c95ff0-c706-4a95-827e-f14bbb9dfa3a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.423825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 800/1590] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:29:30 2022] POST /identity/v3/users => generated 290 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:30.424711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c95ff0-c706-4a95-827e-f14bbb9dfa3a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.435932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c95ff0-c706-4a95-827e-f14bbb9dfa3a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.440978 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e8f5ed-99ea-4f38-bd0b-1d5be5ad1484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.441595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e8f5ed-99ea-4f38-bd0b-1d5be5ad1484 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.441997 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e8f5ed-99ea-4f38-bd0b-1d5be5ad1484 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.442275 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e8f5ed-99ea-4f38-bd0b-1d5be5ad1484 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.442925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e8f5ed-99ea-4f38-bd0b-1d5be5ad1484 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.444583 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e8f5ed-99ea-4f38-bd0b-1d5be5ad1484 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.461193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 801/1591] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:30 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:30.465397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 791/1592] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:30 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:30.465542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e8f5ed-99ea-4f38-bd0b-1d5be5ad1484 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.465626 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cd96a0-0cd3-49c0-bba4-c54e503f054a None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.473679 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cd96a0-0cd3-49c0-bba4-c54e503f054a None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.476971 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e8f5ed-99ea-4f38-bd0b-1d5be5ad1484 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.479975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb842afa-8639-4b84-95cc-0edfe31504eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.480589 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb842afa-8639-4b84-95cc-0edfe31504eb admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.480876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb842afa-8639-4b84-95cc-0edfe31504eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.481151 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb842afa-8639-4b84-95cc-0edfe31504eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.481801 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb842afa-8639-4b84-95cc-0edfe31504eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.482374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5441f32-edd0-45f0-af42-a1ff5170b414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.483062 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5441f32-edd0-45f0-af42-a1ff5170b414 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.483419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb842afa-8639-4b84-95cc-0edfe31504eb admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.483564 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5441f32-edd0-45f0-af42-a1ff5170b414 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.483753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5441f32-edd0-45f0-af42-a1ff5170b414 admin admin] PATH_INFO: `/v3/projects/882ef38f39024b8c88016fb4deacdbe8/users/f2a6dab7cbbe48239e0e203ce14566ec/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.495687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 792/1593] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:30 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:30.496598 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb842afa-8639-4b84-95cc-0edfe31504eb admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.504680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5441f32-edd0-45f0-af42-a1ff5170b414 admin admin] RBAC: Authorizing `identity:create_grant(project_id=882ef38f39024b8c88016fb4deacdbe8, user_id=f2a6dab7cbbe48239e0e203ce14566ec, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.507550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb842afa-8639-4b84-95cc-0edfe31504eb admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.509892 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5441f32-edd0-45f0-af42-a1ff5170b414 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.514254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2d93c5-d31a-45ff-84a6-c759a5e9ac02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.514905 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2d93c5-d31a-45ff-84a6-c759a5e9ac02 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.515185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2d93c5-d31a-45ff-84a6-c759a5e9ac02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.515488 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2d93c5-d31a-45ff-84a6-c759a5e9ac02 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.516116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2d93c5-d31a-45ff-84a6-c759a5e9ac02 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.517834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2d93c5-d31a-45ff-84a6-c759a5e9ac02 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.524289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:30.532057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 802/1594] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:30 2022] PUT /identity/v3/projects/882ef38f39024b8c88016fb4deacdbe8/users/f2a6dab7cbbe48239e0e203ce14566ec/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:30.533553 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5441f32-edd0-45f0-af42-a1ff5170b414 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.534063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 793/1595] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:30 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:30.534986 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2d93c5-d31a-45ff-84a6-c759a5e9ac02 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.536842 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7f2d93c5-d31a-45ff-84a6-c759a5e9ac02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:30.554285 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:30.603882 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2d93c5-d31a-45ff-84a6-c759a5e9ac02 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.624196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5441f32-edd0-45f0-af42-a1ff5170b414 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.629554 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9de1d6d-14a3-4e0d-b94a-380904761693 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.630179 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9de1d6d-14a3-4e0d-b94a-380904761693 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.630483 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9de1d6d-14a3-4e0d-b94a-380904761693 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.630756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9de1d6d-14a3-4e0d-b94a-380904761693 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.631411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9de1d6d-14a3-4e0d-b94a-380904761693 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.633871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9de1d6d-14a3-4e0d-b94a-380904761693 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.645742 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 803/1596] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:30 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:30.646695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9de1d6d-14a3-4e0d-b94a-380904761693 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.658461 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9de1d6d-14a3-4e0d-b94a-380904761693 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.663529 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607b4563-8f0e-4d65-b8ef-3a2fdba555f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.664843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607b4563-8f0e-4d65-b8ef-3a2fdba555f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.665125 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607b4563-8f0e-4d65-b8ef-3a2fdba555f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.665397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607b4563-8f0e-4d65-b8ef-3a2fdba555f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.666054 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607b4563-8f0e-4d65-b8ef-3a2fdba555f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.667213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb2aa6b-5c28-468f-87a9-973ce3254fe3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.667715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607b4563-8f0e-4d65-b8ef-3a2fdba555f0 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.668136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb2aa6b-5c28-468f-87a9-973ce3254fe3 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.668610 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb2aa6b-5c28-468f-87a9-973ce3254fe3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.669081 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb2aa6b-5c28-468f-87a9-973ce3254fe3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.680204 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 804/1597] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:30 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:30.680995 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607b4563-8f0e-4d65-b8ef-3a2fdba555f0 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.681926 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb2aa6b-5c28-468f-87a9-973ce3254fe3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.683493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb2aa6b-5c28-468f-87a9-973ce3254fe3 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.691635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607b4563-8f0e-4d65-b8ef-3a2fdba555f0 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.694408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 794/1598] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:30.695969 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea139565-0b04-4ea8-8009-408a1d305902 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.696262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea139565-0b04-4ea8-8009-408a1d305902 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.696572 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea139565-0b04-4ea8-8009-408a1d305902 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.697387 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540bc7fb-d275-489f-882e-7559da89e2ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.698130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 795/1599] 10.222.0.64 () {54 vars in 961 bytes} [Tue Oct 4 14:29:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:29:30.698989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea139565-0b04-4ea8-8009-408a1d305902 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.700591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540bc7fb-d275-489f-882e-7559da89e2ca admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.701576 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540bc7fb-d275-489f-882e-7559da89e2ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.702391 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540bc7fb-d275-489f-882e-7559da89e2ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.703546 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540bc7fb-d275-489f-882e-7559da89e2ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.705591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540bc7fb-d275-489f-882e-7559da89e2ca admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.713854 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea139565-0b04-4ea8-8009-408a1d305902 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.720504 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 805/1600] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:30 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:30.720823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845a11e1-ba0b-4c2a-bbb0-7387766e709f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.721427 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845a11e1-ba0b-4c2a-bbb0-7387766e709f admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.721724 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845a11e1-ba0b-4c2a-bbb0-7387766e709f admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.722008 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845a11e1-ba0b-4c2a-bbb0-7387766e709f admin admin] PATH_INFO: `/v3/projects/1e5731dbee8946a281caefaeb02bd7ff/users/54b3856eb61a487dab21d8ffa345701c/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.723587 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540bc7fb-d275-489f-882e-7559da89e2ca admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.738522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540bc7fb-d275-489f-882e-7559da89e2ca admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.746039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845a11e1-ba0b-4c2a-bbb0-7387766e709f admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e5731dbee8946a281caefaeb02bd7ff, user_id=54b3856eb61a487dab21d8ffa345701c, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.746295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b4995b-e4c8-4378-af71-f775a74379ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.746683 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b4995b-e4c8-4378-af71-f775a74379ca admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.746963 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b4995b-e4c8-4378-af71-f775a74379ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.747267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b4995b-e4c8-4378-af71-f775a74379ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.747917 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b4995b-e4c8-4378-af71-f775a74379ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.749596 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b4995b-e4c8-4378-af71-f775a74379ca admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.750637 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845a11e1-ba0b-4c2a-bbb0-7387766e709f admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.761239 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 806/1601] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:30 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:30.766166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4695dd4-0094-4d1b-84c1-ac1ae0565d59 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.766422 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4695dd4-0094-4d1b-84c1-ac1ae0565d59 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.766697 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4695dd4-0094-4d1b-84c1-ac1ae0565d59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.773260 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 796/1602] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:30 2022] PUT /identity/v3/projects/1e5731dbee8946a281caefaeb02bd7ff/users/54b3856eb61a487dab21d8ffa345701c/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:30.776588 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845a11e1-ba0b-4c2a-bbb0-7387766e709f admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.803934 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-b4695dd4-0094-4d1b-84c1-ac1ae0565d59 None None] MFA Rules not processed for user `f2a6dab7cbbe48239e0e203ce14566ec`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:30.810802 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b4695dd4-0094-4d1b-84c1-ac1ae0565d59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:30.892821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845a11e1-ba0b-4c2a-bbb0-7387766e709f admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.900162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c8e579-e804-4fc2-8af9-60faadd41998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.901015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c8e579-e804-4fc2-8af9-60faadd41998 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.901336 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c8e579-e804-4fc2-8af9-60faadd41998 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.901578 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c8e579-e804-4fc2-8af9-60faadd41998 admin admin] PATH_INFO: `/v3/projects/641b76e704ad40ce86b9e2ce749e9494/users/18865210e0d948aea088af8a089a8dea/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.917809 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 807/1603] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:30 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:30.918714 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4695dd4-0094-4d1b-84c1-ac1ae0565d59 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.931189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4695dd4-0094-4d1b-84c1-ac1ae0565d59 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:30.932529 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c8e579-e804-4fc2-8af9-60faadd41998 admin admin] RBAC: Authorizing `identity:create_grant(project_id=641b76e704ad40ce86b9e2ce749e9494, user_id=18865210e0d948aea088af8a089a8dea, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.937212 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8407d826-d6fe-43b6-96e1-bd08109db238 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:30.937516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c8e579-e804-4fc2-8af9-60faadd41998 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.938372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8407d826-d6fe-43b6-96e1-bd08109db238 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.938805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8407d826-d6fe-43b6-96e1-bd08109db238 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.939300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8407d826-d6fe-43b6-96e1-bd08109db238 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.940139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8407d826-d6fe-43b6-96e1-bd08109db238 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:30.942252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8407d826-d6fe-43b6-96e1-bd08109db238 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:30.961967 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 808/1604] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:30 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:30.963830 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1c7a14-6b1e-4fb9-8514-a56ebf9fdab5 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:30.964217 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1c7a14-6b1e-4fb9-8514-a56ebf9fdab5 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:30.965266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1c7a14-6b1e-4fb9-8514-a56ebf9fdab5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:30.965382 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 797/1605] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:30 2022] PUT /identity/v3/projects/641b76e704ad40ce86b9e2ce749e9494/users/18865210e0d948aea088af8a089a8dea/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:30.965710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c8e579-e804-4fc2-8af9-60faadd41998 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:30.995215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:31.018023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-6a1c7a14-6b1e-4fb9-8514-a56ebf9fdab5 None None] MFA Rules not processed for user `03081d7d581745c88c12e9d089fcb2d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:31.018889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6a1c7a14-6b1e-4fb9-8514-a56ebf9fdab5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:31.064788 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c8e579-e804-4fc2-8af9-60faadd41998 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:31.068968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:31.070803 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41d27e4-e002-45fb-8da4-0188d1fd65ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:31.071614 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41d27e4-e002-45fb-8da4-0188d1fd65ef admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:31.072009 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41d27e4-e002-45fb-8da4-0188d1fd65ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:31.072258 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41d27e4-e002-45fb-8da4-0188d1fd65ef admin admin] PATH_INFO: `/v3/projects/64993786e21b45beb7829ceaede4777a/users/c5b38375d6ff4d0db0f3f7bc89ca8798/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:31.094320 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41d27e4-e002-45fb-8da4-0188d1fd65ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=64993786e21b45beb7829ceaede4777a, user_id=c5b38375d6ff4d0db0f3f7bc89ca8798, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:31.104648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41d27e4-e002-45fb-8da4-0188d1fd65ef admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:31.125839 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 798/1606] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:30 2022] PUT /identity/v3/projects/64993786e21b45beb7829ceaede4777a/users/c5b38375d6ff4d0db0f3f7bc89ca8798/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:31.130009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 809/1607] 10.222.0.64 () {58 vars in 1058 bytes} [Tue Oct 4 14:29:30 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:31.130009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56036783-dbb0-49d9-81c9-9a5c5f8aac87 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:31.130599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56036783-dbb0-49d9-81c9-9a5c5f8aac87 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:31.131263 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56036783-dbb0-49d9-81c9-9a5c5f8aac87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:31.136278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41d27e4-e002-45fb-8da4-0188d1fd65ef admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:31.181303 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-56036783-dbb0-49d9-81c9-9a5c5f8aac87 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:31.183790 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-56036783-dbb0-49d9-81c9-9a5c5f8aac87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:31.232011 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41d27e4-e002-45fb-8da4-0188d1fd65ef admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:31.238728 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7aed0a-5d67-4016-9e75-d5e07a4f8191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:31.239500 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7aed0a-5d67-4016-9e75-d5e07a4f8191 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:31.239758 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7aed0a-5d67-4016-9e75-d5e07a4f8191 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:31.240196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7aed0a-5d67-4016-9e75-d5e07a4f8191 admin admin] PATH_INFO: `/v3/projects/3a4fa5d0aa0e4bf98816e4600023ca99/users/fc02bb4b8a32400faef8743d2418a648/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:31.273181 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7aed0a-5d67-4016-9e75-d5e07a4f8191 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a4fa5d0aa0e4bf98816e4600023ca99, user_id=fc02bb4b8a32400faef8743d2418a648, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:31.279350 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7aed0a-5d67-4016-9e75-d5e07a4f8191 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:31.282170 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 810/1608] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:31.286427 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e67fe5b-e459-43d4-85d5-451b1b0cfa27 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:31.287823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e67fe5b-e459-43d4-85d5-451b1b0cfa27 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:31.288580 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e67fe5b-e459-43d4-85d5-451b1b0cfa27 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:31.292605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 811/1609] 10.222.0.64 () {54 vars in 961 bytes} [Tue Oct 4 14:29:31 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:29:31.294092 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e67fe5b-e459-43d4-85d5-451b1b0cfa27 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:31.307810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 799/1610] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:31 2022] PUT /identity/v3/projects/3a4fa5d0aa0e4bf98816e4600023ca99/users/fc02bb4b8a32400faef8743d2418a648/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:31.309685 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86038864-2f83-42a2-91a3-16bf76e36720 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:31.309869 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86038864-2f83-42a2-91a3-16bf76e36720 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:31.310256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86038864-2f83-42a2-91a3-16bf76e36720 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:31.345059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-86038864-2f83-42a2-91a3-16bf76e36720 None None] MFA Rules not processed for user `18865210e0d948aea088af8a089a8dea`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:31.346710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-86038864-2f83-42a2-91a3-16bf76e36720 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:31.444813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e67fe5b-e459-43d4-85d5-451b1b0cfa27 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:31.462529 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0cb6a4-72a4-49d1-b171-c05718ea47d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:31.464133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0cb6a4-72a4-49d1-b171-c05718ea47d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:31.464610 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0cb6a4-72a4-49d1-b171-c05718ea47d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:31.465082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0cb6a4-72a4-49d1-b171-c05718ea47d4 admin admin] PATH_INFO: `/v3/projects/da10281be41c43988e60ecfafd5d0d12/users/3f6d20541ee8417ba718190c88c48deb/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:31.478414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 800/1611] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:31 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:31.480967 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0665b9-973e-4f77-bd98-de4a919dcdb4 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:31.481601 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0665b9-973e-4f77-bd98-de4a919dcdb4 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:31.482170 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0665b9-973e-4f77-bd98-de4a919dcdb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:31.487063 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0cb6a4-72a4-49d1-b171-c05718ea47d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=da10281be41c43988e60ecfafd5d0d12, user_id=3f6d20541ee8417ba718190c88c48deb, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:31.491231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0cb6a4-72a4-49d1-b171-c05718ea47d4 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:31.517449 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 812/1612] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:31 2022] PUT /identity/v3/projects/da10281be41c43988e60ecfafd5d0d12/users/3f6d20541ee8417ba718190c88c48deb/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:31.518462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0cb6a4-72a4-49d1-b171-c05718ea47d4 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:31.525934 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-6e0665b9-973e-4f77-bd98-de4a919dcdb4 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:31.528864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-6e0665b9-973e-4f77-bd98-de4a919dcdb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:31.545291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:31.602294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:31.630397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0cb6a4-72a4-49d1-b171-c05718ea47d4 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:31.642062 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e89301c-b3f8-4be3-adeb-b5a5195556f4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:31.642956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e89301c-b3f8-4be3-adeb-b5a5195556f4 service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:31.643842 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e89301c-b3f8-4be3-adeb-b5a5195556f4 service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:31.643842 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e89301c-b3f8-4be3-adeb-b5a5195556f4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:31.657784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 801/1613] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:31.660607 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127a671e-14b1-4f28-9a9e-4fa0a77db285 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:31.660809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127a671e-14b1-4f28-9a9e-4fa0a77db285 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:31.661210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127a671e-14b1-4f28-9a9e-4fa0a77db285 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:31.703012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-127a671e-14b1-4f28-9a9e-4fa0a77db285 None None] MFA Rules not processed for user `03081d7d581745c88c12e9d089fcb2d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:31.705493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-127a671e-14b1-4f28-9a9e-4fa0a77db285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:31.720861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 802/1614] 10.222.0.64 () {58 vars in 1058 bytes} [Tue Oct 4 14:29:31 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:31.722232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127a671e-14b1-4f28-9a9e-4fa0a77db285 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:31.761722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e89301c-b3f8-4be3-adeb-b5a5195556f4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:31.765002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e89301c-b3f8-4be3-adeb-b5a5195556f4 service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:31.780273 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 813/1615] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:31 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 262 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:31.782062 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b213f360-9cb6-447f-8c1c-14de9b714679 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:31.782677 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b213f360-9cb6-447f-8c1c-14de9b714679 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:31.783296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b213f360-9cb6-447f-8c1c-14de9b714679 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:31.809368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-b213f360-9cb6-447f-8c1c-14de9b714679 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:31.810852 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b213f360-9cb6-447f-8c1c-14de9b714679 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:31.821066 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 814/1616] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:31.822066 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b213f360-9cb6-447f-8c1c-14de9b714679 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:31.828273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127a671e-14b1-4f28-9a9e-4fa0a77db285 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:31.847897 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e5885e-c1bf-4656-b183-af8244535bfe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:31.848190 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e5885e-c1bf-4656-b183-af8244535bfe service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:31.848590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e5885e-c1bf-4656-b183-af8244535bfe service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:31.848964 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e5885e-c1bf-4656-b183-af8244535bfe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:31.849144 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b213f360-9cb6-447f-8c1c-14de9b714679 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:31.866015 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abf2b43-32d5-4ab5-8abd-2aa285073324 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:31.868136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abf2b43-32d5-4ab5-8abd-2aa285073324 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:31.869070 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abf2b43-32d5-4ab5-8abd-2aa285073324 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:31.869529 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abf2b43-32d5-4ab5-8abd-2aa285073324 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:31.870380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abf2b43-32d5-4ab5-8abd-2aa285073324 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:31.873139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abf2b43-32d5-4ab5-8abd-2aa285073324 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:31.880909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e5885e-c1bf-4656-b183-af8244535bfe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:31.882660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e5885e-c1bf-4656-b183-af8244535bfe service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:31.917289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 815/1617] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:29:31 2022] POST /identity/v3/projects => generated 369 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:31.918760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 803/1618] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:31.919743 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e5885e-c1bf-4656-b183-af8244535bfe service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:31.920148 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3100a93c-10ba-4e24-8bc1-bcb5a75fa77e None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:31.920815 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3100a93c-10ba-4e24-8bc1-bcb5a75fa77e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:31.921247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3100a93c-10ba-4e24-8bc1-bcb5a75fa77e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:31.944947 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e5885e-c1bf-4656-b183-af8244535bfe service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:31.947681 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-3100a93c-10ba-4e24-8bc1-bcb5a75fa77e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:31.950408 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-3100a93c-10ba-4e24-8bc1-bcb5a75fa77e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:32.059471 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95ceb60-9fe2-4ba4-88d3-993a64fdc723 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:32.059843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95ceb60-9fe2-4ba4-88d3-993a64fdc723 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:32.060128 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95ceb60-9fe2-4ba4-88d3-993a64fdc723 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:32.060553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95ceb60-9fe2-4ba4-88d3-993a64fdc723 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:32.115004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 816/1619] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:32.115922 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3100a93c-10ba-4e24-8bc1-bcb5a75fa77e None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:32.235094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95ceb60-9fe2-4ba4-88d3-993a64fdc723 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:32.237809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95ceb60-9fe2-4ba4-88d3-993a64fdc723 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:32.262448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:32.283961 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 804/1620] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:32.285594 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95ceb60-9fe2-4ba4-88d3-993a64fdc723 service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:32.286049 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:32.302888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3100a93c-10ba-4e24-8bc1-bcb5a75fa77e None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:32.306718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95ceb60-9fe2-4ba4-88d3-993a64fdc723 service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:32.313535 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfe342f-db3d-4edd-9d38-9e326f8fcc89 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:32.314829 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfe342f-db3d-4edd-9d38-9e326f8fcc89 service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:32.315867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfe342f-db3d-4edd-9d38-9e326f8fcc89 service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:32.316594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfe342f-db3d-4edd-9d38-9e326f8fcc89 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:32.318339 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5c3962-f89b-4b86-9277-48a4932fa367 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:32.318989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5c3962-f89b-4b86-9277-48a4932fa367 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:32.319273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5c3962-f89b-4b86-9277-48a4932fa367 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:32.319547 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5c3962-f89b-4b86-9277-48a4932fa367 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:32.341563 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5c3962-f89b-4b86-9277-48a4932fa367 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:32.343044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5c3962-f89b-4b86-9277-48a4932fa367 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:32.360671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 805/1621] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 76 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:32.361536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5c3962-f89b-4b86-9277-48a4932fa367 service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:32.386678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5c3962-f89b-4b86-9277-48a4932fa367 service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:32.402336 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb7df5d-f024-47d5-bae0-3769d727ab01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:32.402989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb7df5d-f024-47d5-bae0-3769d727ab01 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:32.403342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb7df5d-f024-47d5-bae0-3769d727ab01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:32.403616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb7df5d-f024-47d5-bae0-3769d727ab01 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:32.404273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb7df5d-f024-47d5-bae0-3769d727ab01 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:32.406300 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb7df5d-f024-47d5-bae0-3769d727ab01 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:32.478908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 806/1622] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:29:32 2022] POST /identity/v3/users => generated 291 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:32.484195 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102ac7b6-218d-41a2-a918-d35267dbad91 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:32.487987 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102ac7b6-218d-41a2-a918-d35267dbad91 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:32.488673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102ac7b6-218d-41a2-a918-d35267dbad91 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:32.493027 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 807/1623] 10.222.0.64 () {54 vars in 912 bytes} [Tue Oct 4 14:29:32 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:29:32.496599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102ac7b6-218d-41a2-a918-d35267dbad91 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:32.507277 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfe342f-db3d-4edd-9d38-9e326f8fcc89 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:32.517878 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfe342f-db3d-4edd-9d38-9e326f8fcc89 service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:32.521550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102ac7b6-218d-41a2-a918-d35267dbad91 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:32.533430 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7678778d-3092-46c2-a58d-1e61f87c4935 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:32.534098 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7678778d-3092-46c2-a58d-1e61f87c4935 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:32.535257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7678778d-3092-46c2-a58d-1e61f87c4935 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:32.536165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7678778d-3092-46c2-a58d-1e61f87c4935 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:32.551455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 817/1624] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:32 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:32.556136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfe342f-db3d-4edd-9d38-9e326f8fcc89 service nova] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:32.584716 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7678778d-3092-46c2-a58d-1e61f87c4935 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:32.594762 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7678778d-3092-46c2-a58d-1e61f87c4935 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:32.601543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfe342f-db3d-4edd-9d38-9e326f8fcc89 service nova] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:32.621641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 808/1625] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:32.623085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdeeb13-d709-49a5-bb37-8312db3232ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:32.624123 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdeeb13-d709-49a5-bb37-8312db3232ce admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:32.624593 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdeeb13-d709-49a5-bb37-8312db3232ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:32.625914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdeeb13-d709-49a5-bb37-8312db3232ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:32.627120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3642e6a1-e089-47db-924a-b45ac3aa9dcd None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:32.627411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3642e6a1-e089-47db-924a-b45ac3aa9dcd None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:32.627686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3642e6a1-e089-47db-924a-b45ac3aa9dcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:32.628188 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdeeb13-d709-49a5-bb37-8312db3232ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:32.632247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdeeb13-d709-49a5-bb37-8312db3232ce admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:32.662632 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 818/1626] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:32 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:32.665298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdeeb13-d709-49a5-bb37-8312db3232ce admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:32.683362 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-3642e6a1-e089-47db-924a-b45ac3aa9dcd None None] MFA Rules not processed for user `03081d7d581745c88c12e9d089fcb2d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:32.686354 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-3642e6a1-e089-47db-924a-b45ac3aa9dcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:32.694083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdeeb13-d709-49a5-bb37-8312db3232ce admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:32.709574 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 809/1627] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Oct 4 14:29:32 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:32.711863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b040addd-4469-42ab-95ba-26a2698fb406 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:32.712705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b040addd-4469-42ab-95ba-26a2698fb406 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:32.713149 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b040addd-4469-42ab-95ba-26a2698fb406 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:32.713442 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b040addd-4469-42ab-95ba-26a2698fb406 admin admin] PATH_INFO: `/v3/projects/ef0d49107f6d45c68af1aeeb817d3fa3/users/029bb92cfe45498aac2aa09bf6d7585e/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:32.751996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3642e6a1-e089-47db-924a-b45ac3aa9dcd None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:32.760592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b040addd-4469-42ab-95ba-26a2698fb406 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef0d49107f6d45c68af1aeeb817d3fa3, user_id=029bb92cfe45498aac2aa09bf6d7585e, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:32.767762 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b040addd-4469-42ab-95ba-26a2698fb406 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:32.803764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 819/1628] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:32 2022] PUT /identity/v3/projects/ef0d49107f6d45c68af1aeeb817d3fa3/users/029bb92cfe45498aac2aa09bf6d7585e/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:32.806019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b040addd-4469-42ab-95ba-26a2698fb406 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:33.070920 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3642e6a1-e089-47db-924a-b45ac3aa9dcd None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:33.090741 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b040addd-4469-42ab-95ba-26a2698fb406 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:33.114509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e942410c-9542-4d43-a66b-3de4da31a1bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:33.115366 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e942410c-9542-4d43-a66b-3de4da31a1bf admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:33.117585 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e942410c-9542-4d43-a66b-3de4da31a1bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:33.118094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e942410c-9542-4d43-a66b-3de4da31a1bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:33.119110 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e942410c-9542-4d43-a66b-3de4da31a1bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:33.121201 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e942410c-9542-4d43-a66b-3de4da31a1bf admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:33.135747 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 820/1629] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:32 2022] GET /identity/v3/roles => generated 1420 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:33.143230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e942410c-9542-4d43-a66b-3de4da31a1bf admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:33.168946 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e942410c-9542-4d43-a66b-3de4da31a1bf admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:33.181225 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be36c1b5-11d7-4911-81e0-15b717a8d8ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:33.185498 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be36c1b5-11d7-4911-81e0-15b717a8d8ef admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:33.187787 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be36c1b5-11d7-4911-81e0-15b717a8d8ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:33.188429 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be36c1b5-11d7-4911-81e0-15b717a8d8ef admin admin] PATH_INFO: `/v3/projects/ef0d49107f6d45c68af1aeeb817d3fa3/users/029bb92cfe45498aac2aa09bf6d7585e/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:33.199202 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be36c1b5-11d7-4911-81e0-15b717a8d8ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef0d49107f6d45c68af1aeeb817d3fa3, user_id=029bb92cfe45498aac2aa09bf6d7585e, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:33.202289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be36c1b5-11d7-4911-81e0-15b717a8d8ef admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:33.232818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 821/1630] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:33 2022] PUT /identity/v3/projects/ef0d49107f6d45c68af1aeeb817d3fa3/users/029bb92cfe45498aac2aa09bf6d7585e/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:33.253298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28340f3-52ad-4b6d-a12b-e47a482d35d1 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:33.255775 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28340f3-52ad-4b6d-a12b-e47a482d35d1 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:33.256818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28340f3-52ad-4b6d-a12b-e47a482d35d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:33.265295 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8a6b58-57dd-451b-ae70-3f997cdab0cf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:33.265978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8a6b58-57dd-451b-ae70-3f997cdab0cf service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:33.266278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8a6b58-57dd-451b-ae70-3f997cdab0cf service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:33.266523 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8a6b58-57dd-451b-ae70-3f997cdab0cf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:33.327741 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-f28340f3-52ad-4b6d-a12b-e47a482d35d1 None None] MFA Rules not processed for user `029bb92cfe45498aac2aa09bf6d7585e`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:33.328998 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f28340f3-52ad-4b6d-a12b-e47a482d35d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:33.427362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:33.499592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 822/1631] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:33 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:33.500763 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a7cda5-77de-4eeb-8c89-eb6935e514f4 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:33.500763 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a7cda5-77de-4eeb-8c89-eb6935e514f4 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:33.501012 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a7cda5-77de-4eeb-8c89-eb6935e514f4 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:33.504766 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 823/1632] 10.222.0.64 () {54 vars in 912 bytes} [Tue Oct 4 14:29:33 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:29:33.511761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5360da24-720e-48f2-b89b-aa7188e94c4b None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:33.512030 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5360da24-720e-48f2-b89b-aa7188e94c4b None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:33.512450 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5360da24-720e-48f2-b89b-aa7188e94c4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:33.562604 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8a6b58-57dd-451b-ae70-3f997cdab0cf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:33.566329 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8a6b58-57dd-451b-ae70-3f997cdab0cf service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:33.569888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-5360da24-720e-48f2-b89b-aa7188e94c4b None None] MFA Rules not processed for user `03081d7d581745c88c12e9d089fcb2d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:33.575302 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5360da24-720e-48f2-b89b-aa7188e94c4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:33.595591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 810/1633] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 844 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:29:33.597092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8a6b58-57dd-451b-ae70-3f997cdab0cf service cinder] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:33.615067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8a6b58-57dd-451b-ae70-3f997cdab0cf service cinder] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:33.622621 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:33.852250 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fb4057-8345-4e82-8a42-b994f3a95e33 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:33.853181 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fb4057-8345-4e82-8a42-b994f3a95e33 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:33.853483 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fb4057-8345-4e82-8a42-b994f3a95e33 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:33.853686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fb4057-8345-4e82-8a42-b994f3a95e33 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:33.873382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 824/1634] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Oct 4 14:29:33 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 357 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:29:33.905421 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fb4057-8345-4e82-8a42-b994f3a95e33 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:33.907701 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fb4057-8345-4e82-8a42-b994f3a95e33 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:33.967601 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 811/1635] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:33 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:34.070800 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5360da24-720e-48f2-b89b-aa7188e94c4b None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:34.088540 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5360da24-720e-48f2-b89b-aa7188e94c4b None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:34.101180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344816cc-60c6-4387-9408-359fb5309dbc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:34.102127 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344816cc-60c6-4387-9408-359fb5309dbc service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:34.102481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344816cc-60c6-4387-9408-359fb5309dbc service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:34.102848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344816cc-60c6-4387-9408-359fb5309dbc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:34.120064 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344816cc-60c6-4387-9408-359fb5309dbc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:34.121769 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344816cc-60c6-4387-9408-359fb5309dbc service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:34.161008 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 825/1636] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:34.827158 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fb4057-8345-4e82-8a42-b994f3a95e33 service cinder] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:34.846117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fb4057-8345-4e82-8a42-b994f3a95e33 service cinder] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:34.859867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9fa756-0f3d-419f-beb4-cf70b0e4e111 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:34.862376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9fa756-0f3d-419f-beb4-cf70b0e4e111 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:34.863402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9fa756-0f3d-419f-beb4-cf70b0e4e111 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:34.868118 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9fa756-0f3d-419f-beb4-cf70b0e4e111 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:35.005074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9fa756-0f3d-419f-beb4-cf70b0e4e111 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:35.007115 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9fa756-0f3d-419f-beb4-cf70b0e4e111 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:35.038462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 812/1637] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:34 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:38.112844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344816cc-60c6-4387-9408-359fb5309dbc service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:38.310641 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344816cc-60c6-4387-9408-359fb5309dbc service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:38.317560 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bf1168-6a0d-4850-9fa3-ea87e0100264 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:38.318754 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bf1168-6a0d-4850-9fa3-ea87e0100264 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:38.319341 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bf1168-6a0d-4850-9fa3-ea87e0100264 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:38.320576 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bf1168-6a0d-4850-9fa3-ea87e0100264 admin admin] PATH_INFO: `/v3/users/f2a6dab7cbbe48239e0e203ce14566ec` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:38.322717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31bf1168-6a0d-4850-9fa3-ea87e0100264 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f2a6dab7cbbe48239e0e203ce14566ec)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:38.326696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: WARNING py.warnings [None req-31bf1168-6a0d-4850-9fa3-ea87e0100264 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 14:29:38.326696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: warnings.warn(msg) Oct 04 14:29:38.326696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: Oct 04 14:29:38.329376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31bf1168-6a0d-4850-9fa3-ea87e0100264 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:38.620588 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-31bf1168-6a0d-4850-9fa3-ea87e0100264 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f2a6dab7cbbe48239e0e203ce14566ec'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:38.647465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-31bf1168-6a0d-4850-9fa3-ea87e0100264 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f2a6dab7cbbe48239e0e203ce14566ec'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:38.653613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:38.745667 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-31bf1168-6a0d-4850-9fa3-ea87e0100264 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f2a6dab7cbbe48239e0e203ce14566ec'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:38.842252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-31bf1168-6a0d-4850-9fa3-ea87e0100264 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f2a6dab7cbbe48239e0e203ce14566ec'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:38.846582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-31bf1168-6a0d-4850-9fa3-ea87e0100264 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f2a6dab7cbbe48239e0e203ce14566ec'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:38.880358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 826/1638] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:29:38 2022] DELETE /identity/v3/users/f2a6dab7cbbe48239e0e203ce14566ec => generated 0 bytes in 767 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:38.885855 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08ee896-05eb-4ad1-a1d2-f0dc99a63087 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:38.886446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08ee896-05eb-4ad1-a1d2-f0dc99a63087 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:38.886446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08ee896-05eb-4ad1-a1d2-f0dc99a63087 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:38.974884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-a08ee896-05eb-4ad1-a1d2-f0dc99a63087 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:38.983544 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a08ee896-05eb-4ad1-a1d2-f0dc99a63087 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:39.144519 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 813/1639] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:29:39.192449 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bf1168-6a0d-4850-9fa3-ea87e0100264 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:39.194413 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-31bf1168-6a0d-4850-9fa3-ea87e0100264 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:39.465177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bf1168-6a0d-4850-9fa3-ea87e0100264 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:39.519020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c15916-f884-4a93-b56f-f972825adde3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:39.519675 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c15916-f884-4a93-b56f-f972825adde3 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:39.519969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c15916-f884-4a93-b56f-f972825adde3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:39.520243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c15916-f884-4a93-b56f-f972825adde3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:39.540629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c15916-f884-4a93-b56f-f972825adde3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:39.542550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c15916-f884-4a93-b56f-f972825adde3 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:39.588763 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 827/1640] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:40.745871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08ee896-05eb-4ad1-a1d2-f0dc99a63087 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:40.747719 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a08ee896-05eb-4ad1-a1d2-f0dc99a63087 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:40.768026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08ee896-05eb-4ad1-a1d2-f0dc99a63087 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:40.774714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7f695b-87fa-4da1-a164-06eae1264779 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:40.775727 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7f695b-87fa-4da1-a164-06eae1264779 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:40.776123 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7f695b-87fa-4da1-a164-06eae1264779 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:40.776548 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7f695b-87fa-4da1-a164-06eae1264779 admin admin] PATH_INFO: `/v3/users/18865210e0d948aea088af8a089a8dea` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:40.778361 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7f695b-87fa-4da1-a164-06eae1264779 admin admin] RBAC: Authorizing `identity:delete_user(user_id=18865210e0d948aea088af8a089a8dea)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:40.783175 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7f695b-87fa-4da1-a164-06eae1264779 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:40.977584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:40.985055 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0a7f695b-87fa-4da1-a164-06eae1264779 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '18865210e0d948aea088af8a089a8dea'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:40.988912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0a7f695b-87fa-4da1-a164-06eae1264779 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '18865210e0d948aea088af8a089a8dea'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:40.997025 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0a7f695b-87fa-4da1-a164-06eae1264779 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '18865210e0d948aea088af8a089a8dea'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:41.021455 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0a7f695b-87fa-4da1-a164-06eae1264779 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '18865210e0d948aea088af8a089a8dea'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:41.076253 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0a7f695b-87fa-4da1-a164-06eae1264779 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '18865210e0d948aea088af8a089a8dea'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:41.113576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 814/1641] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:29:40 2022] DELETE /identity/v3/users/18865210e0d948aea088af8a089a8dea => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:41.126090 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac15e569-fc0d-458e-8963-5a5fcccba96d None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:41.126677 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac15e569-fc0d-458e-8963-5a5fcccba96d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:41.126873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac15e569-fc0d-458e-8963-5a5fcccba96d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:41.201515 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-ac15e569-fc0d-458e-8963-5a5fcccba96d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:41.208110 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ac15e569-fc0d-458e-8963-5a5fcccba96d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:41.390810 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 828/1642] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:29:41.500299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7f695b-87fa-4da1-a164-06eae1264779 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:41.502140 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0a7f695b-87fa-4da1-a164-06eae1264779 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:41.701892 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7f695b-87fa-4da1-a164-06eae1264779 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:41.717295 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be389f2a-6d62-49a9-9011-259417827ebb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:41.717999 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be389f2a-6d62-49a9-9011-259417827ebb service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:41.718289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be389f2a-6d62-49a9-9011-259417827ebb service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:41.718568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be389f2a-6d62-49a9-9011-259417827ebb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:41.743155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be389f2a-6d62-49a9-9011-259417827ebb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:41.745179 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be389f2a-6d62-49a9-9011-259417827ebb service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:41.775989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 815/1643] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 276 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:43.029125 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac15e569-fc0d-458e-8963-5a5fcccba96d None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:43.042242 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ac15e569-fc0d-458e-8963-5a5fcccba96d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:43.128074 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac15e569-fc0d-458e-8963-5a5fcccba96d None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:43.175801 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:43.182386 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:43.183122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:43.188082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9 admin admin] PATH_INFO: `/v3/projects/641b76e704ad40ce86b9e2ce749e9494` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:43.196091 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be389f2a-6d62-49a9-9011-259417827ebb service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:43.199704 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-be389f2a-6d62-49a9-9011-259417827ebb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:43.208962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=641b76e704ad40ce86b9e2ce749e9494)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:43.225497 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:43.284656 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be389f2a-6d62-49a9-9011-259417827ebb service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:43.361588 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae6113f-440e-441f-98fe-f767bbe4a192 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:43.364599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae6113f-440e-441f-98fe-f767bbe4a192 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:43.364891 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae6113f-440e-441f-98fe-f767bbe4a192 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:43.365179 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae6113f-440e-441f-98fe-f767bbe4a192 admin admin] PATH_INFO: `/v3/projects/882ef38f39024b8c88016fb4deacdbe8` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:43.375237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae6113f-440e-441f-98fe-f767bbe4a192 admin admin] RBAC: Authorizing `identity:delete_project(project_id=882ef38f39024b8c88016fb4deacdbe8)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:43.379225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: WARNING py.warnings [None req-3ae6113f-440e-441f-98fe-f767bbe4a192 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 14:29:43.379225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: warnings.warn(msg) Oct 04 14:29:43.379225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: Oct 04 14:29:43.390393 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae6113f-440e-441f-98fe-f767bbe4a192 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:43.470262 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '641b76e704ad40ce86b9e2ce749e9494', 'request_id': 'req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:43.485831 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '641b76e704ad40ce86b9e2ce749e9494', 'request_id': 'req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:43.497526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '641b76e704ad40ce86b9e2ce749e9494', 'request_id': 'req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:43.515682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:43.543305 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9 admin admin] The token cache is being invalidate because project 641b76e704ad40ce86b9e2ce749e9494 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:29:43.543899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:43.545495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:43.547666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 829/1644] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:29:43 2022] DELETE /identity/v3/projects/641b76e704ad40ce86b9e2ce749e9494 => generated 0 bytes in 519 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:43.554080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:43.560007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:43.692704 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3ae6113f-440e-441f-98fe-f767bbe4a192 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '882ef38f39024b8c88016fb4deacdbe8', 'request_id': 'req-3ae6113f-440e-441f-98fe-f767bbe4a192'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:43.729035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3ae6113f-440e-441f-98fe-f767bbe4a192 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '882ef38f39024b8c88016fb4deacdbe8', 'request_id': 'req-3ae6113f-440e-441f-98fe-f767bbe4a192'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:43.744143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3ae6113f-440e-441f-98fe-f767bbe4a192 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '882ef38f39024b8c88016fb4deacdbe8', 'request_id': 'req-3ae6113f-440e-441f-98fe-f767bbe4a192'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:43.777797 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3ae6113f-440e-441f-98fe-f767bbe4a192 admin admin] The token cache is being invalidate because project 882ef38f39024b8c88016fb4deacdbe8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:29:43.778091 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3ae6113f-440e-441f-98fe-f767bbe4a192 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:43.779166 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3ae6113f-440e-441f-98fe-f767bbe4a192 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:43.784639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 816/1645] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:29:43 2022] DELETE /identity/v3/projects/882ef38f39024b8c88016fb4deacdbe8 => generated 0 bytes in 590 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:43.785802 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae6113f-440e-441f-98fe-f767bbe4a192 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:43.796499 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-3ae6113f-440e-441f-98fe-f767bbe4a192 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:43.899606 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:43.935702 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae6113f-440e-441f-98fe-f767bbe4a192 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:43.950998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592f2d66-13c6-499d-9da1-0e35208b57f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:43.951670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592f2d66-13c6-499d-9da1-0e35208b57f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:43.951953 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592f2d66-13c6-499d-9da1-0e35208b57f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:43.952225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592f2d66-13c6-499d-9da1-0e35208b57f5 admin admin] PATH_INFO: `/v3/users/a99a904626e44182a020368a63c4dad3` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:43.955986 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592f2d66-13c6-499d-9da1-0e35208b57f5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a99a904626e44182a020368a63c4dad3)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:43.957988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592f2d66-13c6-499d-9da1-0e35208b57f5 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:43.976653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c5ea65-bdc5-4000-bf18-3c15c2cdf9b9 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:44.017598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9211a739-5388-44fa-8db8-22d722c1bc82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:44.020980 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9211a739-5388-44fa-8db8-22d722c1bc82 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:44.024645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9211a739-5388-44fa-8db8-22d722c1bc82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:44.025419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9211a739-5388-44fa-8db8-22d722c1bc82 admin admin] PATH_INFO: `/v3/users/029bb92cfe45498aac2aa09bf6d7585e` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:44.030376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9211a739-5388-44fa-8db8-22d722c1bc82 admin admin] RBAC: Authorizing `identity:delete_user(user_id=029bb92cfe45498aac2aa09bf6d7585e)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:44.039676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9211a739-5388-44fa-8db8-22d722c1bc82 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:44.069405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-592f2d66-13c6-499d-9da1-0e35208b57f5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a99a904626e44182a020368a63c4dad3'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:44.070988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-592f2d66-13c6-499d-9da1-0e35208b57f5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a99a904626e44182a020368a63c4dad3'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:44.072720 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-592f2d66-13c6-499d-9da1-0e35208b57f5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a99a904626e44182a020368a63c4dad3'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:44.088425 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-592f2d66-13c6-499d-9da1-0e35208b57f5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a99a904626e44182a020368a63c4dad3'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:44.120730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-592f2d66-13c6-499d-9da1-0e35208b57f5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a99a904626e44182a020368a63c4dad3'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:44.155909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 817/1646] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:29:43 2022] DELETE /identity/v3/users/a99a904626e44182a020368a63c4dad3 => generated 0 bytes in 370 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:44.163174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912170a4-81f8-474c-91b6-ba2029019288 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:44.164400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912170a4-81f8-474c-91b6-ba2029019288 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:44.164400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912170a4-81f8-474c-91b6-ba2029019288 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:44.196863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9211a739-5388-44fa-8db8-22d722c1bc82 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '029bb92cfe45498aac2aa09bf6d7585e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:44.211950 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9211a739-5388-44fa-8db8-22d722c1bc82 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '029bb92cfe45498aac2aa09bf6d7585e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:44.226621 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-912170a4-81f8-474c-91b6-ba2029019288 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:44.228355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-912170a4-81f8-474c-91b6-ba2029019288 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:44.228499 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9211a739-5388-44fa-8db8-22d722c1bc82 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '029bb92cfe45498aac2aa09bf6d7585e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:44.250701 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9211a739-5388-44fa-8db8-22d722c1bc82 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '029bb92cfe45498aac2aa09bf6d7585e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:44.251840 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9211a739-5388-44fa-8db8-22d722c1bc82 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '029bb92cfe45498aac2aa09bf6d7585e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:44.285744 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 830/1647] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:29:43 2022] DELETE /identity/v3/users/029bb92cfe45498aac2aa09bf6d7585e => generated 0 bytes in 729 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:44.286561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d9bea-e396-4974-938e-ff89ee8d02d1 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:44.286739 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d9bea-e396-4974-938e-ff89ee8d02d1 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:44.286966 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d9bea-e396-4974-938e-ff89ee8d02d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:44.351425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-738d9bea-e396-4974-938e-ff89ee8d02d1 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:44.358854 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-738d9bea-e396-4974-938e-ff89ee8d02d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:44.415953 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 818/1648] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:44.424008 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1e2990-5270-40d9-919b-222311063fea None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:44.424303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1e2990-5270-40d9-919b-222311063fea None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:44.424607 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1e2990-5270-40d9-919b-222311063fea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:44.483414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-7d1e2990-5270-40d9-919b-222311063fea None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:44.484949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7d1e2990-5270-40d9-919b-222311063fea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:44.500954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 831/1649] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:44.511558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 819/1650] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:29:44.516124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738d9bea-e396-4974-938e-ff89ee8d02d1 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:44.518734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1e2990-5270-40d9-919b-222311063fea None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:44.520608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7d1e2990-5270-40d9-919b-222311063fea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:44.535966 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738d9bea-e396-4974-938e-ff89ee8d02d1 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:44.549342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61974ca-7e5b-4eb0-a786-49d5525ce623 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:44.549657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61974ca-7e5b-4eb0-a786-49d5525ce623 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:44.549913 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61974ca-7e5b-4eb0-a786-49d5525ce623 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:44.550189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61974ca-7e5b-4eb0-a786-49d5525ce623 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:44.550862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61974ca-7e5b-4eb0-a786-49d5525ce623 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:29:44.551506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61974ca-7e5b-4eb0-a786-49d5525ce623 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:44.565216 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61974ca-7e5b-4eb0-a786-49d5525ce623 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:44.602005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 832/1651] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:29:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:29:44.606192 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61974ca-7e5b-4eb0-a786-49d5525ce623 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:44.616412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:44.622909 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61974ca-7e5b-4eb0-a786-49d5525ce623 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:44.650749 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98823bd-9bcc-4276-bbf0-3f8c57b5dd91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:44.651495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98823bd-9bcc-4276-bbf0-3f8c57b5dd91 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:44.651845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98823bd-9bcc-4276-bbf0-3f8c57b5dd91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:44.652132 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98823bd-9bcc-4276-bbf0-3f8c57b5dd91 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:44.660960 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98823bd-9bcc-4276-bbf0-3f8c57b5dd91 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:44.665879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98823bd-9bcc-4276-bbf0-3f8c57b5dd91 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:44.722253 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1e2990-5270-40d9-919b-222311063fea None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:44.734495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35622a5d-c875-4df6-9c29-7186ca1fe8c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:44.735123 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35622a5d-c875-4df6-9c29-7186ca1fe8c9 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:44.735408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35622a5d-c875-4df6-9c29-7186ca1fe8c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:44.735802 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35622a5d-c875-4df6-9c29-7186ca1fe8c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:44.745679 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 833/1652] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:29:44 2022] POST /identity/v3/projects => generated 371 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:29:44.748854 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98823bd-9bcc-4276-bbf0-3f8c57b5dd91 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:44.764213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35622a5d-c875-4df6-9c29-7186ca1fe8c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:44.768640 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35622a5d-c875-4df6-9c29-7186ca1fe8c9 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:44.800392 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 820/1653] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 283 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:29:44.931649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98823bd-9bcc-4276-bbf0-3f8c57b5dd91 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:44.938068 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de4cf0c-16fc-437a-9e0f-6f2736fcb3dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:44.938762 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de4cf0c-16fc-437a-9e0f-6f2736fcb3dc admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:44.939045 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de4cf0c-16fc-437a-9e0f-6f2736fcb3dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:44.939316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de4cf0c-16fc-437a-9e0f-6f2736fcb3dc admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:44.940014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de4cf0c-16fc-437a-9e0f-6f2736fcb3dc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:44.942712 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de4cf0c-16fc-437a-9e0f-6f2736fcb3dc admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:45.013514 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 834/1654] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:29:44 2022] POST /identity/v3/users => generated 286 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:45.017581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35622a5d-c875-4df6-9c29-7186ca1fe8c9 service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:45.024586 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:45.050247 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35622a5d-c875-4df6-9c29-7186ca1fe8c9 service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:45.065578 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e418d849-d1de-4acf-ab68-2ad451911c63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:45.066227 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e418d849-d1de-4acf-ab68-2ad451911c63 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:45.066511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e418d849-d1de-4acf-ab68-2ad451911c63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:45.066787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e418d849-d1de-4acf-ab68-2ad451911c63 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:45.067445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e418d849-d1de-4acf-ab68-2ad451911c63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:45.069130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e418d849-d1de-4acf-ab68-2ad451911c63 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:45.099419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 821/1655] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:45 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:45.100665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de4cf0c-16fc-437a-9e0f-6f2736fcb3dc admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:45.121296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de4cf0c-16fc-437a-9e0f-6f2736fcb3dc admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:45.128083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff95ef3f-c2b2-4ce2-8fa7-e272add6b26d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:45.129920 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff95ef3f-c2b2-4ce2-8fa7-e272add6b26d admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:45.130859 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff95ef3f-c2b2-4ce2-8fa7-e272add6b26d admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:45.130859 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff95ef3f-c2b2-4ce2-8fa7-e272add6b26d admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:45.131791 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff95ef3f-c2b2-4ce2-8fa7-e272add6b26d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:45.139010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff95ef3f-c2b2-4ce2-8fa7-e272add6b26d admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:45.158767 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 835/1656] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:45 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:29:45.162687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e418d849-d1de-4acf-ab68-2ad451911c63 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:45.175062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e418d849-d1de-4acf-ab68-2ad451911c63 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:45.181275 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808c3abf-2d65-4d38-b642-ecc2d9a9808c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:45.182113 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808c3abf-2d65-4d38-b642-ecc2d9a9808c admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:45.182342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808c3abf-2d65-4d38-b642-ecc2d9a9808c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:45.182624 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808c3abf-2d65-4d38-b642-ecc2d9a9808c admin admin] PATH_INFO: `/v3/projects/2e3d973693cc45d0abaa391442971e88/users/3f421d1a9ba04b188a753b862a2b5c7b/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:45.201597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808c3abf-2d65-4d38-b642-ecc2d9a9808c admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e3d973693cc45d0abaa391442971e88, user_id=3f421d1a9ba04b188a753b862a2b5c7b, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:45.204786 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808c3abf-2d65-4d38-b642-ecc2d9a9808c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:45.241234 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580372cd-43e4-4412-8dd9-c8df96411340 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:45.242583 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580372cd-43e4-4412-8dd9-c8df96411340 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:45.243046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580372cd-43e4-4412-8dd9-c8df96411340 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:45.246633 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 822/1657] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:45 2022] PUT /identity/v3/projects/2e3d973693cc45d0abaa391442971e88/users/3f421d1a9ba04b188a753b862a2b5c7b/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:45.281571 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-580372cd-43e4-4412-8dd9-c8df96411340 None None] MFA Rules not processed for user `3f421d1a9ba04b188a753b862a2b5c7b`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:45.283896 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-580372cd-43e4-4412-8dd9-c8df96411340 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:45.469765 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 836/1658] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:45 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:45.476859 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808c3abf-2d65-4d38-b642-ecc2d9a9808c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:45.674256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808c3abf-2d65-4d38-b642-ecc2d9a9808c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:45.700995 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a248674-05e3-418a-8b11-cc0e96117f0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:45.702234 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a248674-05e3-418a-8b11-cc0e96117f0c admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:45.702723 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a248674-05e3-418a-8b11-cc0e96117f0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:45.703419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580372cd-43e4-4412-8dd9-c8df96411340 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:45.703790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a248674-05e3-418a-8b11-cc0e96117f0c admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:45.705318 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a248674-05e3-418a-8b11-cc0e96117f0c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:45.705552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-580372cd-43e4-4412-8dd9-c8df96411340 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:45.721960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a248674-05e3-418a-8b11-cc0e96117f0c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:45.753856 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580372cd-43e4-4412-8dd9-c8df96411340 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:45.774730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e35a7a-cd36-4dab-9245-c757a5bc2b18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:45.775744 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e35a7a-cd36-4dab-9245-c757a5bc2b18 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:45.776208 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e35a7a-cd36-4dab-9245-c757a5bc2b18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:45.776696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e35a7a-cd36-4dab-9245-c757a5bc2b18 admin admin] PATH_INFO: `/v3/projects/ef0d49107f6d45c68af1aeeb817d3fa3` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:45.778692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e35a7a-cd36-4dab-9245-c757a5bc2b18 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ef0d49107f6d45c68af1aeeb817d3fa3)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:45.782194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e35a7a-cd36-4dab-9245-c757a5bc2b18 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:45.783903 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 823/1659] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:29:45 2022] POST /identity/v3/projects => generated 373 bytes in 308 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:45.790379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a248674-05e3-418a-8b11-cc0e96117f0c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:45.908215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:45.918652 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-61e35a7a-cd36-4dab-9245-c757a5bc2b18 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ef0d49107f6d45c68af1aeeb817d3fa3', 'request_id': 'req-61e35a7a-cd36-4dab-9245-c757a5bc2b18'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:45.936736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-61e35a7a-cd36-4dab-9245-c757a5bc2b18 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ef0d49107f6d45c68af1aeeb817d3fa3', 'request_id': 'req-61e35a7a-cd36-4dab-9245-c757a5bc2b18'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:45.944301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-61e35a7a-cd36-4dab-9245-c757a5bc2b18 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ef0d49107f6d45c68af1aeeb817d3fa3', 'request_id': 'req-61e35a7a-cd36-4dab-9245-c757a5bc2b18'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:45.963302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a248674-05e3-418a-8b11-cc0e96117f0c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:45.970647 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-61e35a7a-cd36-4dab-9245-c757a5bc2b18 admin admin] The token cache is being invalidate because project ef0d49107f6d45c68af1aeeb817d3fa3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:29:45.971142 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-61e35a7a-cd36-4dab-9245-c757a5bc2b18 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:45.971999 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-61e35a7a-cd36-4dab-9245-c757a5bc2b18 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:45.977704 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 837/1660] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:29:45 2022] DELETE /identity/v3/projects/ef0d49107f6d45c68af1aeeb817d3fa3 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:45.979570 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad0ca9c-6e71-4083-a6f7-b8f132bf24fc None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:45.979820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad0ca9c-6e71-4083-a6f7-b8f132bf24fc None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:45.980102 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad0ca9c-6e71-4083-a6f7-b8f132bf24fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:46.003974 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-3ad0ca9c-6e71-4083-a6f7-b8f132bf24fc None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:46.005337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-3ad0ca9c-6e71-4083-a6f7-b8f132bf24fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:46.107322 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96489a7-fe67-4426-bba8-cc69d22fee0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:46.107959 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96489a7-fe67-4426-bba8-cc69d22fee0b admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:46.108250 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96489a7-fe67-4426-bba8-cc69d22fee0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:46.109020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96489a7-fe67-4426-bba8-cc69d22fee0b admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:46.109207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96489a7-fe67-4426-bba8-cc69d22fee0b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:46.111215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96489a7-fe67-4426-bba8-cc69d22fee0b admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:46.134577 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:46.154631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 838/1661] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:46.156256 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5c37ff-d0f0-4f7c-b6e6-22eb1ab150c3 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:46.156624 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5c37ff-d0f0-4f7c-b6e6-22eb1ab150c3 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:46.157205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5c37ff-d0f0-4f7c-b6e6-22eb1ab150c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:46.171433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 824/1662] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:29:45 2022] POST /identity/v3/users => generated 293 bytes in 383 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:46.177791 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96489a7-fe67-4426-bba8-cc69d22fee0b admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:46.182556 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d96489a7-fe67-4426-bba8-cc69d22fee0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:46.213253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-ce5c37ff-d0f0-4f7c-b6e6-22eb1ab150c3 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:46.214612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ce5c37ff-d0f0-4f7c-b6e6-22eb1ab150c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:46.216749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96489a7-fe67-4426-bba8-cc69d22fee0b admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:46.228271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18d9cab-6542-4ba2-b438-0dd66996588d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:46.228958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18d9cab-6542-4ba2-b438-0dd66996588d admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:46.230555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18d9cab-6542-4ba2-b438-0dd66996588d admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:46.230864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18d9cab-6542-4ba2-b438-0dd66996588d admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:46.231562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18d9cab-6542-4ba2-b438-0dd66996588d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:46.234271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18d9cab-6542-4ba2-b438-0dd66996588d admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:46.237643 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 839/1663] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:46.241993 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5c37ff-d0f0-4f7c-b6e6-22eb1ab150c3 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:46.244671 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ce5c37ff-d0f0-4f7c-b6e6-22eb1ab150c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:46.292204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 825/1664] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:46 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:46.293959 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18d9cab-6542-4ba2-b438-0dd66996588d admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:46.307458 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5c37ff-d0f0-4f7c-b6e6-22eb1ab150c3 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:46.343349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18d9cab-6542-4ba2-b438-0dd66996588d admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:46.345509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87635d0-cdd6-4b38-a3be-3798c724dabb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:46.346353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87635d0-cdd6-4b38-a3be-3798c724dabb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:46.346747 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87635d0-cdd6-4b38-a3be-3798c724dabb admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:46.347226 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87635d0-cdd6-4b38-a3be-3798c724dabb admin admin] PATH_INFO: `/v3/projects/0f752b93233147d59c19b3dc7f5b42c4` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:46.349784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87635d0-cdd6-4b38-a3be-3798c724dabb admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f752b93233147d59c19b3dc7f5b42c4)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:46.354093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87635d0-cdd6-4b38-a3be-3798c724dabb admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:46.371746 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f823e7-930a-484c-9640-332e4bd724f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:46.372841 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f823e7-930a-484c-9640-332e4bd724f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:46.373341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f823e7-930a-484c-9640-332e4bd724f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:46.373835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f823e7-930a-484c-9640-332e4bd724f1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:46.375007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f823e7-930a-484c-9640-332e4bd724f1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:29:46.375529 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f823e7-930a-484c-9640-332e4bd724f1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:46.378767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f823e7-930a-484c-9640-332e4bd724f1 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:46.442853 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 826/1665] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:29:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:29:46.445878 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cee4e1e-2b33-4114-ad0d-d8ebecce3fa7 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:46.446375 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cee4e1e-2b33-4114-ad0d-d8ebecce3fa7 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:46.448947 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cee4e1e-2b33-4114-ad0d-d8ebecce3fa7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:46.515005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-f87635d0-cdd6-4b38-a3be-3798c724dabb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f752b93233147d59c19b3dc7f5b42c4', 'request_id': 'req-f87635d0-cdd6-4b38-a3be-3798c724dabb'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:46.551517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-f87635d0-cdd6-4b38-a3be-3798c724dabb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f752b93233147d59c19b3dc7f5b42c4', 'request_id': 'req-f87635d0-cdd6-4b38-a3be-3798c724dabb'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:46.583716 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-7cee4e1e-2b33-4114-ad0d-d8ebecce3fa7 None None] MFA Rules not processed for user `54b3856eb61a487dab21d8ffa345701c`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:46.588002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7cee4e1e-2b33-4114-ad0d-d8ebecce3fa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:46.610563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-f87635d0-cdd6-4b38-a3be-3798c724dabb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f752b93233147d59c19b3dc7f5b42c4', 'request_id': 'req-f87635d0-cdd6-4b38-a3be-3798c724dabb'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:46.656958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-f87635d0-cdd6-4b38-a3be-3798c724dabb admin admin] The token cache is being invalidate because project 0f752b93233147d59c19b3dc7f5b42c4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:29:46.661799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-f87635d0-cdd6-4b38-a3be-3798c724dabb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:46.667169 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-f87635d0-cdd6-4b38-a3be-3798c724dabb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:46.684885 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 840/1666] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:29:46 2022] DELETE /identity/v3/projects/0f752b93233147d59c19b3dc7f5b42c4 => generated 0 bytes in 436 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:46.687413 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87635d0-cdd6-4b38-a3be-3798c724dabb admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:46.693183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f87635d0-cdd6-4b38-a3be-3798c724dabb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:46.790609 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 827/1667] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:46 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 347 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:46.792855 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cee4e1e-2b33-4114-ad0d-d8ebecce3fa7 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:46.796553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7cee4e1e-2b33-4114-ad0d-d8ebecce3fa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:46.873353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:46.913064 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87635d0-cdd6-4b38-a3be-3798c724dabb admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:46.925654 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e67800e-6092-4f78-ba59-a3041290d724 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:46.927956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e67800e-6092-4f78-ba59-a3041290d724 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:46.928240 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e67800e-6092-4f78-ba59-a3041290d724 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:46.931148 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e67800e-6092-4f78-ba59-a3041290d724 admin admin] PATH_INFO: `/v3/projects/6841f8c226564d6ab5ff4ac6df9c89c7/users/b9cdd4d2faf248cc85533f990a47fa7c/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:46.980663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e67800e-6092-4f78-ba59-a3041290d724 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6841f8c226564d6ab5ff4ac6df9c89c7, user_id=b9cdd4d2faf248cc85533f990a47fa7c, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:46.987961 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e67800e-6092-4f78-ba59-a3041290d724 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:46.990753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cee4e1e-2b33-4114-ad0d-d8ebecce3fa7 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:47.001518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409bc5ed-e307-4f0e-af98-03bb6507b94c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:47.005489 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409bc5ed-e307-4f0e-af98-03bb6507b94c admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:47.006252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409bc5ed-e307-4f0e-af98-03bb6507b94c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:47.007738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409bc5ed-e307-4f0e-af98-03bb6507b94c admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:47.010560 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409bc5ed-e307-4f0e-af98-03bb6507b94c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:47.016544 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409bc5ed-e307-4f0e-af98-03bb6507b94c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:47.022838 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 841/1668] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:46 2022] PUT /identity/v3/projects/6841f8c226564d6ab5ff4ac6df9c89c7/users/b9cdd4d2faf248cc85533f990a47fa7c/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:47.024645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb46aff6-418e-403e-b9a0-71e41b009996 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:47.024801 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb46aff6-418e-403e-b9a0-71e41b009996 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:47.025027 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb46aff6-418e-403e-b9a0-71e41b009996 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:47.076178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-cb46aff6-418e-403e-b9a0-71e41b009996 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:47.077455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-cb46aff6-418e-403e-b9a0-71e41b009996 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:47.088693 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 828/1669] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:29:46 2022] POST /identity/v3/projects => generated 385 bytes in 294 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:47.088693 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409bc5ed-e307-4f0e-af98-03bb6507b94c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:47.093685 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-409bc5ed-e307-4f0e-af98-03bb6507b94c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:47.184473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 842/1670] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:47.193500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb46aff6-418e-403e-b9a0-71e41b009996 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:47.210602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:47.223814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb46aff6-418e-403e-b9a0-71e41b009996 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:47.251815 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409bc5ed-e307-4f0e-af98-03bb6507b94c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:47.259178 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0df60b-6062-4969-ac04-0fd735658bcd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:47.259985 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0df60b-6062-4969-ac04-0fd735658bcd service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:47.260275 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0df60b-6062-4969-ac04-0fd735658bcd service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:47.260597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0df60b-6062-4969-ac04-0fd735658bcd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:47.364980 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0df60b-6062-4969-ac04-0fd735658bcd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:47.366739 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0df60b-6062-4969-ac04-0fd735658bcd service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:47.376944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882fd166-8bf2-4904-a17c-878b38d073d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:47.378025 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882fd166-8bf2-4904-a17c-878b38d073d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:47.378464 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882fd166-8bf2-4904-a17c-878b38d073d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:47.379997 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 829/1671] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:47 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:47.381763 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0df60b-6062-4969-ac04-0fd735658bcd service nova] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:47.392221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882fd166-8bf2-4904-a17c-878b38d073d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:47.395712 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882fd166-8bf2-4904-a17c-878b38d073d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:47.399376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0df60b-6062-4969-ac04-0fd735658bcd service nova] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:47.405643 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882fd166-8bf2-4904-a17c-878b38d073d5 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:47.406430 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac93e46-ca61-461c-bb61-bd733ac959f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:47.407386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac93e46-ca61-461c-bb61-bd733ac959f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:47.409997 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac93e46-ca61-461c-bb61-bd733ac959f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:47.410322 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac93e46-ca61-461c-bb61-bd733ac959f9 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:47.410998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac93e46-ca61-461c-bb61-bd733ac959f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:47.415137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac93e46-ca61-461c-bb61-bd733ac959f9 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:47.435828 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 843/1672] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:47 2022] GET /identity/v3/roles => generated 1420 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:47.439447 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39685243-02d8-4d7d-a243-4a4adaac5b6a None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:47.439994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39685243-02d8-4d7d-a243-4a4adaac5b6a None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:47.441463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39685243-02d8-4d7d-a243-4a4adaac5b6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:47.465639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 830/1673] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:29:47 2022] POST /identity/v3/users => generated 293 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:47.466540 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac93e46-ca61-461c-bb61-bd733ac959f9 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:47.481233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac93e46-ca61-461c-bb61-bd733ac959f9 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:47.488668 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4d88bd-2ae3-4cad-a3a1-110507cf8400 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:47.489452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4d88bd-2ae3-4cad-a3a1-110507cf8400 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:47.489831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4d88bd-2ae3-4cad-a3a1-110507cf8400 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:47.490118 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4d88bd-2ae3-4cad-a3a1-110507cf8400 admin admin] PATH_INFO: `/v3/projects/6841f8c226564d6ab5ff4ac6df9c89c7/users/b9cdd4d2faf248cc85533f990a47fa7c/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:47.493336 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4d88bd-2ae3-4cad-a3a1-110507cf8400 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6841f8c226564d6ab5ff4ac6df9c89c7, user_id=b9cdd4d2faf248cc85533f990a47fa7c, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:47.496523 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4d88bd-2ae3-4cad-a3a1-110507cf8400 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:47.514920 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-39685243-02d8-4d7d-a243-4a4adaac5b6a None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:47.515629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 831/1674] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:47 2022] PUT /identity/v3/projects/6841f8c226564d6ab5ff4ac6df9c89c7/users/b9cdd4d2faf248cc85533f990a47fa7c/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:47.516882 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4d88bd-2ae3-4cad-a3a1-110507cf8400 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:47.517418 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-39685243-02d8-4d7d-a243-4a4adaac5b6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:47.682560 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 844/1675] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:47.688786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64312cf4-5921-46cd-b90c-496b83236fc5 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:47.689066 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64312cf4-5921-46cd-b90c-496b83236fc5 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:47.689342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64312cf4-5921-46cd-b90c-496b83236fc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:47.728869 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4d88bd-2ae3-4cad-a3a1-110507cf8400 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:47.746381 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029ca6da-1a76-42c3-843e-e9dda2eb3766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:47.747038 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ca6da-1a76-42c3-843e-e9dda2eb3766 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:47.747324 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ca6da-1a76-42c3-843e-e9dda2eb3766 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:47.747597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ca6da-1a76-42c3-843e-e9dda2eb3766 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:47.748266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029ca6da-1a76-42c3-843e-e9dda2eb3766 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:47.752383 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:47.757043 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029ca6da-1a76-42c3-843e-e9dda2eb3766 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:47.772134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-64312cf4-5921-46cd-b90c-496b83236fc5 None None] MFA Rules not processed for user `b9cdd4d2faf248cc85533f990a47fa7c`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:47.774179 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-64312cf4-5921-46cd-b90c-496b83236fc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:47.802691 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 832/1676] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:47 2022] GET /identity/v3/roles => generated 1420 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:47.806412 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029ca6da-1a76-42c3-843e-e9dda2eb3766 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:47.833828 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029ca6da-1a76-42c3-843e-e9dda2eb3766 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:47.844396 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649b01a7-a7f1-40d6-bc47-33653fb05b12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:47.845002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649b01a7-a7f1-40d6-bc47-33653fb05b12 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:47.845285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649b01a7-a7f1-40d6-bc47-33653fb05b12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:47.845558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649b01a7-a7f1-40d6-bc47-33653fb05b12 admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:47.849834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649b01a7-a7f1-40d6-bc47-33653fb05b12 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:29:47.850156 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649b01a7-a7f1-40d6-bc47-33653fb05b12 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:47.852629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649b01a7-a7f1-40d6-bc47-33653fb05b12 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:47.892028 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 833/1677] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:29:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:29:47.893183 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649b01a7-a7f1-40d6-bc47-33653fb05b12 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:47.919614 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 845/1678] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:47 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:47.920617 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64312cf4-5921-46cd-b90c-496b83236fc5 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:47.926675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649b01a7-a7f1-40d6-bc47-33653fb05b12 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:47.939544 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30aa9f7-f468-4e89-a106-ccef6a814592 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:47.940583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30aa9f7-f468-4e89-a106-ccef6a814592 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:47.940869 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30aa9f7-f468-4e89-a106-ccef6a814592 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:47.941151 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30aa9f7-f468-4e89-a106-ccef6a814592 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:47.941313 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64312cf4-5921-46cd-b90c-496b83236fc5 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:47.941815 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30aa9f7-f468-4e89-a106-ccef6a814592 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:47.943451 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30aa9f7-f468-4e89-a106-ccef6a814592 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:47.957231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80582cb-1ae9-4026-a79e-c2e35c9ae194 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:47.957612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80582cb-1ae9-4026-a79e-c2e35c9ae194 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:47.957949 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80582cb-1ae9-4026-a79e-c2e35c9ae194 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:47.958086 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80582cb-1ae9-4026-a79e-c2e35c9ae194 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:47.959677 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80582cb-1ae9-4026-a79e-c2e35c9ae194 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:47.962770 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80582cb-1ae9-4026-a79e-c2e35c9ae194 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:47.970680 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 834/1679] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:47 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:47.979094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30aa9f7-f468-4e89-a106-ccef6a814592 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:47.981521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d30aa9f7-f468-4e89-a106-ccef6a814592 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:48.037039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:48.073004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 846/1680] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:29:47 2022] POST /identity/v3/projects => generated 405 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:48.073836 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80582cb-1ae9-4026-a79e-c2e35c9ae194 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:48.191231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30aa9f7-f468-4e89-a106-ccef6a814592 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:48.248049 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80582cb-1ae9-4026-a79e-c2e35c9ae194 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:48.279838 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba71099-0115-40d3-9b30-c5f4c1a84a12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:48.287746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba71099-0115-40d3-9b30-c5f4c1a84a12 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:48.287746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba71099-0115-40d3-9b30-c5f4c1a84a12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:48.287746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba71099-0115-40d3-9b30-c5f4c1a84a12 admin admin] PATH_INFO: `/v3/projects/65e045e4ee9c43b99b001eb2c6196f41/users/7945a41633134ef0ad654cd0e70183a5/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:48.333657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba71099-0115-40d3-9b30-c5f4c1a84a12 admin admin] RBAC: Authorizing `identity:create_grant(project_id=65e045e4ee9c43b99b001eb2c6196f41, user_id=7945a41633134ef0ad654cd0e70183a5, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:48.337434 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba71099-0115-40d3-9b30-c5f4c1a84a12 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:48.391557 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5767c604-7501-49c7-88cd-bbad0c46ce09 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:48.392499 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5767c604-7501-49c7-88cd-bbad0c46ce09 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:48.396825 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5767c604-7501-49c7-88cd-bbad0c46ce09 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:48.397303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5767c604-7501-49c7-88cd-bbad0c46ce09 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:48.407008 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 847/1681] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:48 2022] PUT /identity/v3/projects/65e045e4ee9c43b99b001eb2c6196f41/users/7945a41633134ef0ad654cd0e70183a5/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:48.407760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba71099-0115-40d3-9b30-c5f4c1a84a12 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:48.589881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5767c604-7501-49c7-88cd-bbad0c46ce09 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:48.591370 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5767c604-7501-49c7-88cd-bbad0c46ce09 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:48.618422 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 835/1682] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:47 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 647 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:48.625792 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aee4a41-e8bd-4376-a593-fec7f050e39a None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:48.626602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aee4a41-e8bd-4376-a593-fec7f050e39a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:48.628593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aee4a41-e8bd-4376-a593-fec7f050e39a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:48.642675 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:48.666929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba71099-0115-40d3-9b30-c5f4c1a84a12 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:48.686744 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86710a76-14d7-44e1-a88c-951762b46cde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:48.687382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86710a76-14d7-44e1-a88c-951762b46cde admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:48.687658 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86710a76-14d7-44e1-a88c-951762b46cde admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:48.691201 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86710a76-14d7-44e1-a88c-951762b46cde admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:48.691850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86710a76-14d7-44e1-a88c-951762b46cde admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:48.701241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86710a76-14d7-44e1-a88c-951762b46cde admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:48.727705 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-3aee4a41-e8bd-4376-a593-fec7f050e39a None None] MFA Rules not processed for user `7945a41633134ef0ad654cd0e70183a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:48.729091 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-3aee4a41-e8bd-4376-a593-fec7f050e39a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:48.796484 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 848/1683] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:29:48 2022] POST /identity/v3/users => generated 303 bytes in 389 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:48.802284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86710a76-14d7-44e1-a88c-951762b46cde admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:48.828958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86710a76-14d7-44e1-a88c-951762b46cde admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:48.847613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71bdfb7-ee24-4d22-8920-2bda3eb27832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:48.848262 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71bdfb7-ee24-4d22-8920-2bda3eb27832 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:48.848848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71bdfb7-ee24-4d22-8920-2bda3eb27832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:48.849386 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71bdfb7-ee24-4d22-8920-2bda3eb27832 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:48.850056 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71bdfb7-ee24-4d22-8920-2bda3eb27832 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:48.851996 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71bdfb7-ee24-4d22-8920-2bda3eb27832 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:48.874851 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 849/1684] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:48 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:48.875845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71bdfb7-ee24-4d22-8920-2bda3eb27832 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:48.965814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 836/1685] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:48 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 345 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:48.970465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aee4a41-e8bd-4376-a593-fec7f050e39a None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:49.005074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aee4a41-e8bd-4376-a593-fec7f050e39a None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:49.019612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71bdfb7-ee24-4d22-8920-2bda3eb27832 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:49.024665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15157dc5-aa7b-428b-ba18-71bc5335a1ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:49.025300 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15157dc5-aa7b-428b-ba18-71bc5335a1ab admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:49.025579 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15157dc5-aa7b-428b-ba18-71bc5335a1ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:49.025964 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15157dc5-aa7b-428b-ba18-71bc5335a1ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:49.026641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15157dc5-aa7b-428b-ba18-71bc5335a1ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:49.028287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15157dc5-aa7b-428b-ba18-71bc5335a1ab admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:49.030187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b561a2-acc7-4817-8eb3-8a7b080d584d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:49.030951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b561a2-acc7-4817-8eb3-8a7b080d584d service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:49.031235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b561a2-acc7-4817-8eb3-8a7b080d584d service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:49.031505 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b561a2-acc7-4817-8eb3-8a7b080d584d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:49.034735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-79b561a2-acc7-4817-8eb3-8a7b080d584d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:49.056795 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 837/1686] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:48 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:49.056795 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15157dc5-aa7b-428b-ba18-71bc5335a1ab admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:49.072993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:49.114969 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15157dc5-aa7b-428b-ba18-71bc5335a1ab admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:49.140000 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ce37a9-3852-4999-a7bb-3b7ae4908289 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:49.140681 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ce37a9-3852-4999-a7bb-3b7ae4908289 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:49.140970 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ce37a9-3852-4999-a7bb-3b7ae4908289 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:49.141245 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ce37a9-3852-4999-a7bb-3b7ae4908289 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:49.141890 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ce37a9-3852-4999-a7bb-3b7ae4908289 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:49.154466 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ce37a9-3852-4999-a7bb-3b7ae4908289 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:49.191308 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b561a2-acc7-4817-8eb3-8a7b080d584d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:49.193149 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b561a2-acc7-4817-8eb3-8a7b080d584d service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:49.214303 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 850/1687] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:48 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:49.215442 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b561a2-acc7-4817-8eb3-8a7b080d584d service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:49.219661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 838/1688] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:29:49 2022] POST /identity/v3/projects => generated 385 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:49.221059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8db83be-8481-467d-a658-4d9269dfdfd3 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:49.223108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8db83be-8481-467d-a658-4d9269dfdfd3 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:49.223807 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8db83be-8481-467d-a658-4d9269dfdfd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:49.248920 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b561a2-acc7-4817-8eb3-8a7b080d584d service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:49.323966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-e8db83be-8481-467d-a658-4d9269dfdfd3 None None] MFA Rules not processed for user `2b4c980bac3e444faa0e6f1a1d270cfa`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:49.329361 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-e8db83be-8481-467d-a658-4d9269dfdfd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:49.476668 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b44e88-b246-49a9-86c9-61a57b2af456 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:49.478417 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b44e88-b246-49a9-86c9-61a57b2af456 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:49.478860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b44e88-b246-49a9-86c9-61a57b2af456 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:49.479723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b44e88-b246-49a9-86c9-61a57b2af456 admin admin] PATH_INFO: `/v3/projects/c35e89527b674ddf86d47a9b0d6713b6/users/ddeafbf6a90a401fafa5ea54552727a3/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:49.528192 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b44e88-b246-49a9-86c9-61a57b2af456 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c35e89527b674ddf86d47a9b0d6713b6, user_id=ddeafbf6a90a401fafa5ea54552727a3, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:49.541375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b44e88-b246-49a9-86c9-61a57b2af456 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:49.638210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 851/1689] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:49 2022] PUT /identity/v3/projects/c35e89527b674ddf86d47a9b0d6713b6/users/ddeafbf6a90a401fafa5ea54552727a3/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 423 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:49.645224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b44e88-b246-49a9-86c9-61a57b2af456 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:49.696271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 839/1690] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:49 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 476 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:49.701806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb3e7d8-3cbf-46d5-891a-8b73f7e4ae81 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:49.703172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb3e7d8-3cbf-46d5-891a-8b73f7e4ae81 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:49.703172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb3e7d8-3cbf-46d5-891a-8b73f7e4ae81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:49.765500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:49.902934 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-9cb3e7d8-3cbf-46d5-891a-8b73f7e4ae81 None None] MFA Rules not processed for user `ddeafbf6a90a401fafa5ea54552727a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:49.909754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9cb3e7d8-3cbf-46d5-891a-8b73f7e4ae81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:49.950905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b44e88-b246-49a9-86c9-61a57b2af456 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:49.969981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346788e9-b79f-460d-afff-baabc34a62a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:49.970619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346788e9-b79f-460d-afff-baabc34a62a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:49.970900 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346788e9-b79f-460d-afff-baabc34a62a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:49.971174 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346788e9-b79f-460d-afff-baabc34a62a4 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:49.971935 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346788e9-b79f-460d-afff-baabc34a62a4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:49.973879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346788e9-b79f-460d-afff-baabc34a62a4 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:50.024742 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 852/1691] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:29:49 2022] POST /identity/v3/users => generated 299 bytes in 386 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:50.026203 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c9cc9a-2082-4d7f-8273-7982a64a18f1 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:50.026479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c9cc9a-2082-4d7f-8273-7982a64a18f1 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:50.026753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c9cc9a-2082-4d7f-8273-7982a64a18f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:50.074038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-00c9cc9a-2082-4d7f-8273-7982a64a18f1 None None] MFA Rules not processed for user `2b4c980bac3e444faa0e6f1a1d270cfa`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:50.076923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-00c9cc9a-2082-4d7f-8273-7982a64a18f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:50.130134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 840/1692] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:49 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 433 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:50.131097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb3e7d8-3cbf-46d5-891a-8b73f7e4ae81 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:50.158115 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb3e7d8-3cbf-46d5-891a-8b73f7e4ae81 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:50.174241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa0fd0d-4517-43e1-8e97-ee2327788dc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:50.174343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0fd0d-4517-43e1-8e97-ee2327788dc1 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:50.174569 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0fd0d-4517-43e1-8e97-ee2327788dc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:50.174848 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0fd0d-4517-43e1-8e97-ee2327788dc1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:50.175677 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa0fd0d-4517-43e1-8e97-ee2327788dc1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:50.179615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa0fd0d-4517-43e1-8e97-ee2327788dc1 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:50.211884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 841/1693] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:50 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:50.217319 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa0fd0d-4517-43e1-8e97-ee2327788dc1 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:50.233393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 853/1694] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:50 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:50.235399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c9cc9a-2082-4d7f-8273-7982a64a18f1 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:50.253818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa0fd0d-4517-43e1-8e97-ee2327788dc1 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:50.265170 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c9cc9a-2082-4d7f-8273-7982a64a18f1 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:50.277897 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c93a43-81cb-40e1-81f4-f960c2590fe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:50.278470 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c93a43-81cb-40e1-81f4-f960c2590fe0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:50.278745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c93a43-81cb-40e1-81f4-f960c2590fe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:50.279017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c93a43-81cb-40e1-81f4-f960c2590fe0 admin admin] PATH_INFO: `/v3/projects/d48aa97d9acf49d5a3ead2ac369b0ee7/users/bd8af26a9a304760ba9e16df59b50e3e/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:50.289067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93602e2a-c222-4f37-86dd-ac300ff34c15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:50.290377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93602e2a-c222-4f37-86dd-ac300ff34c15 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:50.290713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93602e2a-c222-4f37-86dd-ac300ff34c15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:50.290994 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93602e2a-c222-4f37-86dd-ac300ff34c15 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:50.292831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93602e2a-c222-4f37-86dd-ac300ff34c15 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:50.297421 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93602e2a-c222-4f37-86dd-ac300ff34c15 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:50.323104 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c93a43-81cb-40e1-81f4-f960c2590fe0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d48aa97d9acf49d5a3ead2ac369b0ee7, user_id=bd8af26a9a304760ba9e16df59b50e3e, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:50.326140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c93a43-81cb-40e1-81f4-f960c2590fe0 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:50.352430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 854/1695] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:50 2022] PUT /identity/v3/projects/d48aa97d9acf49d5a3ead2ac369b0ee7/users/bd8af26a9a304760ba9e16df59b50e3e/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:50.357147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c93a43-81cb-40e1-81f4-f960c2590fe0 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:50.367472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c0c93a43-81cb-40e1-81f4-f960c2590fe0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:50.375061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 842/1696] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:29:50 2022] POST /identity/v3/projects => generated 403 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:50.381163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93602e2a-c222-4f37-86dd-ac300ff34c15 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:50.487926 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:50.593247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c93a43-81cb-40e1-81f4-f960c2590fe0 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:50.634294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:50.645126 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93602e2a-c222-4f37-86dd-ac300ff34c15 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:50.669545 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb274186-7166-491a-9c9f-27fd34427f15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:50.670480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb274186-7166-491a-9c9f-27fd34427f15 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:50.670597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb274186-7166-491a-9c9f-27fd34427f15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:50.674367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb274186-7166-491a-9c9f-27fd34427f15 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:50.675149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb274186-7166-491a-9c9f-27fd34427f15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:50.677102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb274186-7166-491a-9c9f-27fd34427f15 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:50.703459 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 843/1697] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:50 2022] GET /identity/v3/roles => generated 1420 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:50.705247 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb274186-7166-491a-9c9f-27fd34427f15 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:50.745010 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb274186-7166-491a-9c9f-27fd34427f15 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:50.759189 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d832722-44ea-4ed3-99b9-69c9b7ece7e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:50.760441 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d832722-44ea-4ed3-99b9-69c9b7ece7e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:50.760818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d832722-44ea-4ed3-99b9-69c9b7ece7e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:50.761211 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d832722-44ea-4ed3-99b9-69c9b7ece7e6 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:50.764572 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d832722-44ea-4ed3-99b9-69c9b7ece7e6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:50.772017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d832722-44ea-4ed3-99b9-69c9b7ece7e6 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:50.800905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e2bb87-ec30-466f-9e4f-f809b696cef2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:50.806850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e2bb87-ec30-466f-9e4f-f809b696cef2 service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:50.807285 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e2bb87-ec30-466f-9e4f-f809b696cef2 service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:50.809365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e2bb87-ec30-466f-9e4f-f809b696cef2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:50.858788 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 844/1698] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:29:50 2022] POST /identity/v3/users => generated 308 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:50.860510 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d832722-44ea-4ed3-99b9-69c9b7ece7e6 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:50.896870 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d832722-44ea-4ed3-99b9-69c9b7ece7e6 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:50.922454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff02bbd-67d7-4bb4-b8ea-440e0765c44b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:50.923264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff02bbd-67d7-4bb4-b8ea-440e0765c44b admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:50.923652 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff02bbd-67d7-4bb4-b8ea-440e0765c44b admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:50.924003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff02bbd-67d7-4bb4-b8ea-440e0765c44b admin admin] PATH_INFO: `/v3/projects/d48aa97d9acf49d5a3ead2ac369b0ee7/users/bd8af26a9a304760ba9e16df59b50e3e/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:50.930411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff02bbd-67d7-4bb4-b8ea-440e0765c44b admin admin] RBAC: Authorizing `identity:create_grant(project_id=d48aa97d9acf49d5a3ead2ac369b0ee7, user_id=bd8af26a9a304760ba9e16df59b50e3e, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:50.949136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff02bbd-67d7-4bb4-b8ea-440e0765c44b admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:51.016117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 845/1699] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:50 2022] PUT /identity/v3/projects/d48aa97d9acf49d5a3ead2ac369b0ee7/users/bd8af26a9a304760ba9e16df59b50e3e/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:51.017416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff02bbd-67d7-4bb4-b8ea-440e0765c44b admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:51.095015 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e2bb87-ec30-466f-9e4f-f809b696cef2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:51.103335 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e2bb87-ec30-466f-9e4f-f809b696cef2 service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:51.133940 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 855/1700] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:50 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 780 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:51.135876 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23e72df-e846-437a-a9d4-d939dd7a96d2 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:51.138755 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23e72df-e846-437a-a9d4-d939dd7a96d2 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:51.140278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23e72df-e846-437a-a9d4-d939dd7a96d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:51.221264 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-f23e72df-e846-437a-a9d4-d939dd7a96d2 None None] MFA Rules not processed for user `bd8af26a9a304760ba9e16df59b50e3e`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:51.225207 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f23e72df-e846-437a-a9d4-d939dd7a96d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:51.368271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff02bbd-67d7-4bb4-b8ea-440e0765c44b admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:51.403964 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7033296e-7aff-4a21-8e89-0d74fb2ce153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:51.405062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7033296e-7aff-4a21-8e89-0d74fb2ce153 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:51.405062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7033296e-7aff-4a21-8e89-0d74fb2ce153 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:51.405259 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7033296e-7aff-4a21-8e89-0d74fb2ce153 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:51.405809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7033296e-7aff-4a21-8e89-0d74fb2ce153 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:51.407504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7033296e-7aff-4a21-8e89-0d74fb2ce153 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:51.448589 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 846/1701] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:51 2022] GET /identity/v3/roles => generated 1420 bytes in 431 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:51.449556 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7033296e-7aff-4a21-8e89-0d74fb2ce153 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:51.482984 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7033296e-7aff-4a21-8e89-0d74fb2ce153 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:51.492524 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464d4f0a-5a2d-4a85-9033-98a326ec2b63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:51.494452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464d4f0a-5a2d-4a85-9033-98a326ec2b63 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:51.495239 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464d4f0a-5a2d-4a85-9033-98a326ec2b63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:51.495828 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464d4f0a-5a2d-4a85-9033-98a326ec2b63 admin admin] PATH_INFO: `/v3/projects/5098801b8b504794bf2354c639ccf673/users/2c13ff8dcfb243da8661202610217148/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:51.540379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464d4f0a-5a2d-4a85-9033-98a326ec2b63 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5098801b8b504794bf2354c639ccf673, user_id=2c13ff8dcfb243da8661202610217148, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:51.545966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464d4f0a-5a2d-4a85-9033-98a326ec2b63 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:51.548040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 856/1702] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:51 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 413 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:51.562552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23e72df-e846-437a-a9d4-d939dd7a96d2 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:51.577127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 847/1703] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:51 2022] PUT /identity/v3/projects/5098801b8b504794bf2354c639ccf673/users/2c13ff8dcfb243da8661202610217148/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:51.580554 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464d4f0a-5a2d-4a85-9033-98a326ec2b63 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:51.774049 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464d4f0a-5a2d-4a85-9033-98a326ec2b63 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:51.785621 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9b0edc-c7de-48d4-be9e-697341408dcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:51.787120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9b0edc-c7de-48d4-be9e-697341408dcd admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:51.787424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9b0edc-c7de-48d4-be9e-697341408dcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:51.790015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9b0edc-c7de-48d4-be9e-697341408dcd admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:51.790760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9b0edc-c7de-48d4-be9e-697341408dcd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:51.795859 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9b0edc-c7de-48d4-be9e-697341408dcd admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:51.796725 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:51.807559 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:51.839996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 848/1704] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:29:51 2022] GET /identity/v3/roles => generated 1420 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:29:51.855083 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9b0edc-c7de-48d4-be9e-697341408dcd admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:51.873416 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23e72df-e846-437a-a9d4-d939dd7a96d2 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:51.909152 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9b0edc-c7de-48d4-be9e-697341408dcd admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:51.925126 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db724e2-8040-4a92-8ba5-0ea20031811e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:51.926852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db724e2-8040-4a92-8ba5-0ea20031811e admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:51.933978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db724e2-8040-4a92-8ba5-0ea20031811e admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:51.934325 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db724e2-8040-4a92-8ba5-0ea20031811e admin admin] PATH_INFO: `/v3/projects/5098801b8b504794bf2354c639ccf673/users/2c13ff8dcfb243da8661202610217148/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:51.946879 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db724e2-8040-4a92-8ba5-0ea20031811e admin admin] RBAC: Authorizing `identity:create_grant(project_id=5098801b8b504794bf2354c639ccf673, user_id=2c13ff8dcfb243da8661202610217148, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:51.954790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db724e2-8040-4a92-8ba5-0ea20031811e admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:51.981635 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 849/1705] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:29:51 2022] PUT /identity/v3/projects/5098801b8b504794bf2354c639ccf673/users/2c13ff8dcfb243da8661202610217148/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:51.982998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a422836-c4f1-4ec3-91ea-73db8c8b32c2 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:51.983260 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a422836-c4f1-4ec3-91ea-73db8c8b32c2 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:51.983535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a422836-c4f1-4ec3-91ea-73db8c8b32c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:52.036810 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84ed398-7903-409b-bf1f-1bfa8d6358a7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:52.037188 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84ed398-7903-409b-bf1f-1bfa8d6358a7 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:52.037645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84ed398-7903-409b-bf1f-1bfa8d6358a7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:52.038105 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84ed398-7903-409b-bf1f-1bfa8d6358a7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:52.066644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-2a422836-c4f1-4ec3-91ea-73db8c8b32c2 None None] MFA Rules not processed for user `c5b38375d6ff4d0db0f3f7bc89ca8798`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:52.068387 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-2a422836-c4f1-4ec3-91ea-73db8c8b32c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:52.199334 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 850/1706] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:51 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:52.201080 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bf0607-fe62-490d-866a-2c6458c53408 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:52.201365 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bf0607-fe62-490d-866a-2c6458c53408 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:52.201640 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bf0607-fe62-490d-866a-2c6458c53408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:52.240221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84ed398-7903-409b-bf1f-1bfa8d6358a7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:52.244354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84ed398-7903-409b-bf1f-1bfa8d6358a7 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:52.278022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 857/1707] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:51 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 718 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:52.281570 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84ed398-7903-409b-bf1f-1bfa8d6358a7 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:52.310309 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-f1bf0607-fe62-490d-866a-2c6458c53408 None None] MFA Rules not processed for user `2c13ff8dcfb243da8661202610217148`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:52.317369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-f1bf0607-fe62-490d-866a-2c6458c53408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:52.537273 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84ed398-7903-409b-bf1f-1bfa8d6358a7 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:52.571426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 851/1708] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:52 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 371 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:52.572477 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bf0607-fe62-490d-866a-2c6458c53408 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:52.644068 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:52.702614 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8251c683-1302-4cb9-ac0e-846dacb08eb7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:52.703245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8251c683-1302-4cb9-ac0e-846dacb08eb7 service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:52.703522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8251c683-1302-4cb9-ac0e-846dacb08eb7 service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:52.703818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8251c683-1302-4cb9-ac0e-846dacb08eb7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:52.733473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8251c683-1302-4cb9-ac0e-846dacb08eb7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:52.733888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bf0607-fe62-490d-866a-2c6458c53408 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:52.735612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8251c683-1302-4cb9-ac0e-846dacb08eb7 service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:52.750063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d5e1e4-3a3c-4e0b-9f2e-264b3df9d91a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:52.751657 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d5e1e4-3a3c-4e0b-9f2e-264b3df9d91a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:52.751994 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d5e1e4-3a3c-4e0b-9f2e-264b3df9d91a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:52.752433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d5e1e4-3a3c-4e0b-9f2e-264b3df9d91a admin admin] PATH_INFO: `/v3/users/7945a41633134ef0ad654cd0e70183a5` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:52.754693 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d5e1e4-3a3c-4e0b-9f2e-264b3df9d91a admin admin] RBAC: Authorizing `identity:delete_user(user_id=7945a41633134ef0ad654cd0e70183a5)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:52.758683 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 858/1709] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:52 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 480 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:52.759037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d5e1e4-3a3c-4e0b-9f2e-264b3df9d91a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:52.767063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:52.774643 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8251c683-1302-4cb9-ac0e-846dacb08eb7 service nova] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:52.863671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-38d5e1e4-3a3c-4e0b-9f2e-264b3df9d91a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7945a41633134ef0ad654cd0e70183a5'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:52.865932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-38d5e1e4-3a3c-4e0b-9f2e-264b3df9d91a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7945a41633134ef0ad654cd0e70183a5'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:52.868198 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-38d5e1e4-3a3c-4e0b-9f2e-264b3df9d91a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7945a41633134ef0ad654cd0e70183a5'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:52.880522 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-38d5e1e4-3a3c-4e0b-9f2e-264b3df9d91a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7945a41633134ef0ad654cd0e70183a5'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:52.906052 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-38d5e1e4-3a3c-4e0b-9f2e-264b3df9d91a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7945a41633134ef0ad654cd0e70183a5'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:52.960752 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 852/1710] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:29:52 2022] DELETE /identity/v3/users/7945a41633134ef0ad654cd0e70183a5 => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:52.969000 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb42ff1-836c-4d9b-bbf5-9c5fd91fe7a3 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:52.969317 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb42ff1-836c-4d9b-bbf5-9c5fd91fe7a3 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:52.969578 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb42ff1-836c-4d9b-bbf5-9c5fd91fe7a3 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:52.975110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 853/1711] 10.222.0.64 () {54 vars in 912 bytes} [Tue Oct 4 14:29:52 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:29:52.978562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355a5f85-c955-4069-b9e3-07a2b3768eb1 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:52.978857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355a5f85-c955-4069-b9e3-07a2b3768eb1 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:52.979139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355a5f85-c955-4069-b9e3-07a2b3768eb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:52.989663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8251c683-1302-4cb9-ac0e-846dacb08eb7 service nova] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:53.035400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-355a5f85-c955-4069-b9e3-07a2b3768eb1 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:53.037401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-355a5f85-c955-4069-b9e3-07a2b3768eb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:53.153105 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e854324e-ef97-4604-b45e-5bb478c62292 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:53.155322 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e854324e-ef97-4604-b45e-5bb478c62292 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:53.158271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e854324e-ef97-4604-b45e-5bb478c62292 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:53.158805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e854324e-ef97-4604-b45e-5bb478c62292 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:53.169325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-e854324e-ef97-4604-b45e-5bb478c62292 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:53.239459 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 854/1712] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:53.244705 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccf8375-4099-44f3-a1e2-08da3a57c9b2 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:53.247298 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccf8375-4099-44f3-a1e2-08da3a57c9b2 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:53.247884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccf8375-4099-44f3-a1e2-08da3a57c9b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:53.351602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-dccf8375-4099-44f3-a1e2-08da3a57c9b2 None None] MFA Rules not processed for user `8dbeccfb9a194992959f0b341ef0d4a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:53.352955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-dccf8375-4099-44f3-a1e2-08da3a57c9b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:53.369307 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e854324e-ef97-4604-b45e-5bb478c62292 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:53.376976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e854324e-ef97-4604-b45e-5bb478c62292 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:53.402922 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 859/1713] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:52 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 629 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:53.404141 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e854324e-ef97-4604-b45e-5bb478c62292 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:53.406956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-e854324e-ef97-4604-b45e-5bb478c62292 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:53.516868 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:53.546521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 855/1714] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Oct 4 14:29:53 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:53.558475 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e854324e-ef97-4604-b45e-5bb478c62292 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:53.570119 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a87828-58a1-4377-927f-74f1d7a3464d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:53.571603 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a87828-58a1-4377-927f-74f1d7a3464d service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:53.573224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a87828-58a1-4377-927f-74f1d7a3464d service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:53.573224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a87828-58a1-4377-927f-74f1d7a3464d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:53.605876 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a87828-58a1-4377-927f-74f1d7a3464d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:53.606252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccf8375-4099-44f3-a1e2-08da3a57c9b2 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:53.607888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-dccf8375-4099-44f3-a1e2-08da3a57c9b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:53.613735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a87828-58a1-4377-927f-74f1d7a3464d service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:53.623456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccf8375-4099-44f3-a1e2-08da3a57c9b2 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:53.637947 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff614e3b-0150-42e8-acf5-4dd4d0ffdab5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:53.638751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff614e3b-0150-42e8-acf5-4dd4d0ffdab5 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:53.639040 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff614e3b-0150-42e8-acf5-4dd4d0ffdab5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:53.639774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff614e3b-0150-42e8-acf5-4dd4d0ffdab5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:53.656291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 860/1715] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:53.665124 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff614e3b-0150-42e8-acf5-4dd4d0ffdab5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:53.666560 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff614e3b-0150-42e8-acf5-4dd4d0ffdab5 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:53.667506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a87828-58a1-4377-927f-74f1d7a3464d service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:53.677197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-18a87828-58a1-4377-927f-74f1d7a3464d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:53.684606 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 856/1716] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:53 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:29:53.717396 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a87828-58a1-4377-927f-74f1d7a3464d service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:53.737905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1850ec5-7ae8-4a5b-bcc9-05e586690d13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:53.739745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1850ec5-7ae8-4a5b-bcc9-05e586690d13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:53.740589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1850ec5-7ae8-4a5b-bcc9-05e586690d13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:53.740969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1850ec5-7ae8-4a5b-bcc9-05e586690d13 admin admin] PATH_INFO: `/v3/users/ddeafbf6a90a401fafa5ea54552727a3` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:53.743400 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1850ec5-7ae8-4a5b-bcc9-05e586690d13 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ddeafbf6a90a401fafa5ea54552727a3)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:53.762747 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1850ec5-7ae8-4a5b-bcc9-05e586690d13 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:53.901948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e1850ec5-7ae8-4a5b-bcc9-05e586690d13 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ddeafbf6a90a401fafa5ea54552727a3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:53.920752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e1850ec5-7ae8-4a5b-bcc9-05e586690d13 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ddeafbf6a90a401fafa5ea54552727a3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:53.953247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e1850ec5-7ae8-4a5b-bcc9-05e586690d13 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ddeafbf6a90a401fafa5ea54552727a3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:54.006608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e1850ec5-7ae8-4a5b-bcc9-05e586690d13 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ddeafbf6a90a401fafa5ea54552727a3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:54.009639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e1850ec5-7ae8-4a5b-bcc9-05e586690d13 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ddeafbf6a90a401fafa5ea54552727a3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:54.029213 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 861/1717] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:29:53 2022] DELETE /identity/v3/users/ddeafbf6a90a401fafa5ea54552727a3 => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:54.030861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbbf478-e6aa-41ae-abf9-e3ca6416908b None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:54.031530 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbbf478-e6aa-41ae-abf9-e3ca6416908b None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:54.031530 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbbf478-e6aa-41ae-abf9-e3ca6416908b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:54.045723 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:54.074928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-0dbbf478-e6aa-41ae-abf9-e3ca6416908b None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:54.076329 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0dbbf478-e6aa-41ae-abf9-e3ca6416908b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:54.323204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 857/1718] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 293 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:29:54.420097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1850ec5-7ae8-4a5b-bcc9-05e586690d13 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:54.422388 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-e1850ec5-7ae8-4a5b-bcc9-05e586690d13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:54.650790 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1850ec5-7ae8-4a5b-bcc9-05e586690d13 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:54.670833 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119374f7-861b-47f8-843d-22f23f041944 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:54.671899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119374f7-861b-47f8-843d-22f23f041944 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:54.672446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119374f7-861b-47f8-843d-22f23f041944 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:54.672939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119374f7-861b-47f8-843d-22f23f041944 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:54.721976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119374f7-861b-47f8-843d-22f23f041944 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:54.729550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119374f7-861b-47f8-843d-22f23f041944 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:54.771820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 862/1719] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:55.019338 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbbf478-e6aa-41ae-abf9-e3ca6416908b None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:55.020722 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0dbbf478-e6aa-41ae-abf9-e3ca6416908b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:55.068066 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbbf478-e6aa-41ae-abf9-e3ca6416908b None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:55.081295 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec9074b-fb66-4b5c-aa6e-e9390062da6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:55.081755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec9074b-fb66-4b5c-aa6e-e9390062da6d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:55.082313 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec9074b-fb66-4b5c-aa6e-e9390062da6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:55.082313 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec9074b-fb66-4b5c-aa6e-e9390062da6d admin admin] PATH_INFO: `/v3/projects/65e045e4ee9c43b99b001eb2c6196f41` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:55.086470 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec9074b-fb66-4b5c-aa6e-e9390062da6d admin admin] RBAC: Authorizing `identity:delete_project(project_id=65e045e4ee9c43b99b001eb2c6196f41)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:55.094961 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec9074b-fb66-4b5c-aa6e-e9390062da6d admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:55.265913 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-eec9074b-fb66-4b5c-aa6e-e9390062da6d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '65e045e4ee9c43b99b001eb2c6196f41', 'request_id': 'req-eec9074b-fb66-4b5c-aa6e-e9390062da6d'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:55.282021 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-eec9074b-fb66-4b5c-aa6e-e9390062da6d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '65e045e4ee9c43b99b001eb2c6196f41', 'request_id': 'req-eec9074b-fb66-4b5c-aa6e-e9390062da6d'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:55.301013 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-eec9074b-fb66-4b5c-aa6e-e9390062da6d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '65e045e4ee9c43b99b001eb2c6196f41', 'request_id': 'req-eec9074b-fb66-4b5c-aa6e-e9390062da6d'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:55.323201 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-eec9074b-fb66-4b5c-aa6e-e9390062da6d admin admin] The token cache is being invalidate because project 65e045e4ee9c43b99b001eb2c6196f41 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:29:55.323690 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-eec9074b-fb66-4b5c-aa6e-e9390062da6d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:55.324926 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-eec9074b-fb66-4b5c-aa6e-e9390062da6d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:55.327895 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 858/1720] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:29:55 2022] DELETE /identity/v3/projects/65e045e4ee9c43b99b001eb2c6196f41 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:55.332181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119374f7-861b-47f8-843d-22f23f041944 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:55.348870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119374f7-861b-47f8-843d-22f23f041944 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:55.522043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:55.536637 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460accac-7de5-4e8e-b739-ea36fcb4bf42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:55.537294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460accac-7de5-4e8e-b739-ea36fcb4bf42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:55.537575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460accac-7de5-4e8e-b739-ea36fcb4bf42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:55.537848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460accac-7de5-4e8e-b739-ea36fcb4bf42 admin admin] PATH_INFO: `/v3/users/bd8af26a9a304760ba9e16df59b50e3e` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:55.542165 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460accac-7de5-4e8e-b739-ea36fcb4bf42 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bd8af26a9a304760ba9e16df59b50e3e)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:55.544186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460accac-7de5-4e8e-b739-ea36fcb4bf42 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:55.692904 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-460accac-7de5-4e8e-b739-ea36fcb4bf42 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bd8af26a9a304760ba9e16df59b50e3e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:55.713399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-460accac-7de5-4e8e-b739-ea36fcb4bf42 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bd8af26a9a304760ba9e16df59b50e3e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:55.738256 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-460accac-7de5-4e8e-b739-ea36fcb4bf42 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bd8af26a9a304760ba9e16df59b50e3e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:55.774106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-460accac-7de5-4e8e-b739-ea36fcb4bf42 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bd8af26a9a304760ba9e16df59b50e3e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:55.774890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-460accac-7de5-4e8e-b739-ea36fcb4bf42 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bd8af26a9a304760ba9e16df59b50e3e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:55.788830 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 863/1721] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:29:55 2022] DELETE /identity/v3/users/bd8af26a9a304760ba9e16df59b50e3e => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:55.811032 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec9074b-fb66-4b5c-aa6e-e9390062da6d admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:55.819026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-eec9074b-fb66-4b5c-aa6e-e9390062da6d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:55.920579 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f935ba-6173-4402-96c3-998c9e54610c None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:55.922630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f935ba-6173-4402-96c3-998c9e54610c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:55.923124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f935ba-6173-4402-96c3-998c9e54610c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:55.972042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-34f935ba-6173-4402-96c3-998c9e54610c None None] MFA Rules not processed for user `2b4c980bac3e444faa0e6f1a1d270cfa`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:55.975318 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-34f935ba-6173-4402-96c3-998c9e54610c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:56.039496 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec9074b-fb66-4b5c-aa6e-e9390062da6d admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:56.120764 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:56.205170 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 864/1722] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:55 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 285 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:56.244343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42dc4f5e-7a46-4b8a-a804-05e7bc1ec493 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '436deb3d60c54a799700986519b2ef68', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:56.245007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dc4f5e-7a46-4b8a-a804-05e7bc1ec493 service placement] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:56.245288 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dc4f5e-7a46-4b8a-a804-05e7bc1ec493 service placement] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:56.245567 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dc4f5e-7a46-4b8a-a804-05e7bc1ec493 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:56.247702 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-42dc4f5e-7a46-4b8a-a804-05e7bc1ec493 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:56.269531 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42dc4f5e-7a46-4b8a-a804-05e7bc1ec493 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:56.273301 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42dc4f5e-7a46-4b8a-a804-05e7bc1ec493 service placement] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:56.301895 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 859/1723] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:55 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 491 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:56.379116 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d2c3de-f832-4174-b4d9-c40654955e51 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:56.394627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e119854-48d2-4a02-b2ee-a70c293ca24c None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:56.394627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e119854-48d2-4a02-b2ee-a70c293ca24c None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:56.394627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e119854-48d2-4a02-b2ee-a70c293ca24c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:56.399272 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d2c3de-f832-4174-b4d9-c40654955e51 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:56.400918 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d2c3de-f832-4174-b4d9-c40654955e51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:56.442451 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-34d2c3de-f832-4174-b4d9-c40654955e51 None None] MFA Rules not processed for user `54b3856eb61a487dab21d8ffa345701c`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:56.442724 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-8e119854-48d2-4a02-b2ee-a70c293ca24c None None] MFA Rules not processed for user `3f6d20541ee8417ba718190c88c48deb`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:56.444081 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8e119854-48d2-4a02-b2ee-a70c293ca24c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:56.451879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-34d2c3de-f832-4174-b4d9-c40654955e51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:56.623921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 860/1724] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:56 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:29:56.625155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e119854-48d2-4a02-b2ee-a70c293ca24c None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:56.626586 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8e119854-48d2-4a02-b2ee-a70c293ca24c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:56.636543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 865/1725] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:56 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:56.640639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f591fc5-a7cd-4d1a-b826-f7839c7ffeb2 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:56.640925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f591fc5-a7cd-4d1a-b826-f7839c7ffeb2 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:56.646259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f591fc5-a7cd-4d1a-b826-f7839c7ffeb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:56.718776 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-9f591fc5-a7cd-4d1a-b826-f7839c7ffeb2 None None] MFA Rules not processed for user `3f6d20541ee8417ba718190c88c48deb`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:56.727762 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9f591fc5-a7cd-4d1a-b826-f7839c7ffeb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:56.745746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 866/1726] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:56 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:56.833720 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e119854-48d2-4a02-b2ee-a70c293ca24c None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:56.850276 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7af2d5-a598-48b1-a61d-8776e69b56e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:56.852142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7af2d5-a598-48b1-a61d-8776e69b56e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:56.852387 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7af2d5-a598-48b1-a61d-8776e69b56e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:56.852731 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7af2d5-a598-48b1-a61d-8776e69b56e2 admin admin] PATH_INFO: `/v3/projects/c35e89527b674ddf86d47a9b0d6713b6` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:56.857387 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7af2d5-a598-48b1-a61d-8776e69b56e2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c35e89527b674ddf86d47a9b0d6713b6)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:56.860219 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7af2d5-a598-48b1-a61d-8776e69b56e2 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:56.999629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-7c7af2d5-a598-48b1-a61d-8776e69b56e2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c35e89527b674ddf86d47a9b0d6713b6', 'request_id': 'req-7c7af2d5-a598-48b1-a61d-8776e69b56e2'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:57.005243 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:57.016649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-7c7af2d5-a598-48b1-a61d-8776e69b56e2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c35e89527b674ddf86d47a9b0d6713b6', 'request_id': 'req-7c7af2d5-a598-48b1-a61d-8776e69b56e2'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:57.036529 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-7c7af2d5-a598-48b1-a61d-8776e69b56e2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c35e89527b674ddf86d47a9b0d6713b6', 'request_id': 'req-7c7af2d5-a598-48b1-a61d-8776e69b56e2'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:57.062433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-7c7af2d5-a598-48b1-a61d-8776e69b56e2 admin admin] The token cache is being invalidate because project c35e89527b674ddf86d47a9b0d6713b6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:29:57.062841 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-7c7af2d5-a598-48b1-a61d-8776e69b56e2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:57.069971 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-7c7af2d5-a598-48b1-a61d-8776e69b56e2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:57.072178 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 861/1727] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:29:56 2022] DELETE /identity/v3/projects/c35e89527b674ddf86d47a9b0d6713b6 => generated 0 bytes in 447 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:57.076881 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f591fc5-a7cd-4d1a-b826-f7839c7ffeb2 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:57.106597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f591fc5-a7cd-4d1a-b826-f7839c7ffeb2 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:57.175287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:57.377768 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f27337d-ef82-4853-855e-ca50fcde079c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:57.378432 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f27337d-ef82-4853-855e-ca50fcde079c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:57.378709 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f27337d-ef82-4853-855e-ca50fcde079c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:57.378980 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f27337d-ef82-4853-855e-ca50fcde079c admin admin] PATH_INFO: `/v3/users/2c13ff8dcfb243da8661202610217148` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:57.384302 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f27337d-ef82-4853-855e-ca50fcde079c admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c13ff8dcfb243da8661202610217148)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:57.384657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f27337d-ef82-4853-855e-ca50fcde079c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:57.502604 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6f27337d-ef82-4853-855e-ca50fcde079c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c13ff8dcfb243da8661202610217148'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:57.528761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6f27337d-ef82-4853-855e-ca50fcde079c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c13ff8dcfb243da8661202610217148'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:57.549200 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6f27337d-ef82-4853-855e-ca50fcde079c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c13ff8dcfb243da8661202610217148'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:57.550602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c0997b-deca-426a-ac74-83e37a5a7672 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:57.551087 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c0997b-deca-426a-ac74-83e37a5a7672 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:57.551176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c0997b-deca-426a-ac74-83e37a5a7672 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:57.563634 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6f27337d-ef82-4853-855e-ca50fcde079c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c13ff8dcfb243da8661202610217148'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:57.564534 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6f27337d-ef82-4853-855e-ca50fcde079c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c13ff8dcfb243da8661202610217148'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:57.594691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 867/1728] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:29:57 2022] DELETE /identity/v3/users/2c13ff8dcfb243da8661202610217148 => generated 0 bytes in 519 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:57.626068 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-16c0997b-deca-426a-ac74-83e37a5a7672 None None] MFA Rules not processed for user `fc02bb4b8a32400faef8743d2418a648`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:57.627337 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-16c0997b-deca-426a-ac74-83e37a5a7672 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:57.781457 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 862/1729] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:57 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:29:58.361358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f27337d-ef82-4853-855e-ca50fcde079c admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:58.363029 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6f27337d-ef82-4853-855e-ca50fcde079c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:58.585183 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c0997b-deca-426a-ac74-83e37a5a7672 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:58.586720 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-16c0997b-deca-426a-ac74-83e37a5a7672 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:58.587001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f27337d-ef82-4853-855e-ca50fcde079c admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:58.602822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6e26cf-c62f-4234-b910-4de1ed837345 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:58.603505 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6e26cf-c62f-4234-b910-4de1ed837345 service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:58.603789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6e26cf-c62f-4234-b910-4de1ed837345 service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:58.604112 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6e26cf-c62f-4234-b910-4de1ed837345 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:58.608948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7e6e26cf-c62f-4234-b910-4de1ed837345 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:58.713510 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:58.817247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6e26cf-c62f-4234-b910-4de1ed837345 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:58.829117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6e26cf-c62f-4234-b910-4de1ed837345 service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:58.836500 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c0997b-deca-426a-ac74-83e37a5a7672 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:58.862403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e381095-b32d-4ae4-9be5-716be5a7b7ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:58.863430 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e381095-b32d-4ae4-9be5-716be5a7b7ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:58.863779 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e381095-b32d-4ae4-9be5-716be5a7b7ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:58.864291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e381095-b32d-4ae4-9be5-716be5a7b7ce admin admin] PATH_INFO: `/v3/projects/5098801b8b504794bf2354c639ccf673` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:58.865964 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e381095-b32d-4ae4-9be5-716be5a7b7ce admin admin] RBAC: Authorizing `identity:delete_project(project_id=5098801b8b504794bf2354c639ccf673)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:58.869896 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e381095-b32d-4ae4-9be5-716be5a7b7ce admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:58.892571 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 868/1730] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:58 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 523 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:58.900286 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6e26cf-c62f-4234-b910-4de1ed837345 service nova] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:58.907086 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7e6e26cf-c62f-4234-b910-4de1ed837345 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:58.989149 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6e26cf-c62f-4234-b910-4de1ed837345 service nova] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:59.016172 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e883c2-8408-4ec8-8314-b1223629f380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:59.016888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e883c2-8408-4ec8-8314-b1223629f380 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:59.017246 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e883c2-8408-4ec8-8314-b1223629f380 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:59.017643 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e883c2-8408-4ec8-8314-b1223629f380 admin admin] PATH_INFO: `/v3/projects/d48aa97d9acf49d5a3ead2ac369b0ee7` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:59.025430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e883c2-8408-4ec8-8314-b1223629f380 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d48aa97d9acf49d5a3ead2ac369b0ee7)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:59.027369 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e883c2-8408-4ec8-8314-b1223629f380 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:59.094248 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5e381095-b32d-4ae4-9be5-716be5a7b7ce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5098801b8b504794bf2354c639ccf673', 'request_id': 'req-5e381095-b32d-4ae4-9be5-716be5a7b7ce'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:59.137803 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5e381095-b32d-4ae4-9be5-716be5a7b7ce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5098801b8b504794bf2354c639ccf673', 'request_id': 'req-5e381095-b32d-4ae4-9be5-716be5a7b7ce'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:59.163643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5e381095-b32d-4ae4-9be5-716be5a7b7ce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5098801b8b504794bf2354c639ccf673', 'request_id': 'req-5e381095-b32d-4ae4-9be5-716be5a7b7ce'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:59.188519 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b6e883c2-8408-4ec8-8314-b1223629f380 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd48aa97d9acf49d5a3ead2ac369b0ee7', 'request_id': 'req-b6e883c2-8408-4ec8-8314-b1223629f380'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:59.193734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:29:59.203321 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5e381095-b32d-4ae4-9be5-716be5a7b7ce admin admin] The token cache is being invalidate because project 5098801b8b504794bf2354c639ccf673 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:29:59.204009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b6e883c2-8408-4ec8-8314-b1223629f380 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd48aa97d9acf49d5a3ead2ac369b0ee7', 'request_id': 'req-b6e883c2-8408-4ec8-8314-b1223629f380'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:59.204800 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5e381095-b32d-4ae4-9be5-716be5a7b7ce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:59.207404 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5e381095-b32d-4ae4-9be5-716be5a7b7ce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:59.210277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 863/1731] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:29:58 2022] DELETE /identity/v3/projects/5098801b8b504794bf2354c639ccf673 => generated 0 bytes in 626 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:59.217357 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f551d3-9b59-4043-a3cb-d915357aca82 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:59.218056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f551d3-9b59-4043-a3cb-d915357aca82 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:59.219044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f551d3-9b59-4043-a3cb-d915357aca82 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:59.221854 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 864/1732] 10.222.0.64 () {54 vars in 912 bytes} [Tue Oct 4 14:29:59 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:29:59.222974 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b6e883c2-8408-4ec8-8314-b1223629f380 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd48aa97d9acf49d5a3ead2ac369b0ee7', 'request_id': 'req-b6e883c2-8408-4ec8-8314-b1223629f380'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:59.223216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56e4be2-aec5-48c9-b6ec-222e142d4fb5 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:59.228274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56e4be2-aec5-48c9-b6ec-222e142d4fb5 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:59.228675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56e4be2-aec5-48c9-b6ec-222e142d4fb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:59.253201 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b6e883c2-8408-4ec8-8314-b1223629f380 admin admin] The token cache is being invalidate because project d48aa97d9acf49d5a3ead2ac369b0ee7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:29:59.253915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b6e883c2-8408-4ec8-8314-b1223629f380 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:59.255460 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b6e883c2-8408-4ec8-8314-b1223629f380 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:29:59.262138 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 869/1733] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:29:58 2022] DELETE /identity/v3/projects/d48aa97d9acf49d5a3ead2ac369b0ee7 => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:29:59.263514 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba212d9-619f-4c94-8c78-e25104428ae5 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:59.265141 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba212d9-619f-4c94-8c78-e25104428ae5 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:59.265141 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba212d9-619f-4c94-8c78-e25104428ae5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:59.276650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-e56e4be2-aec5-48c9-b6ec-222e142d4fb5 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:59.277951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-e56e4be2-aec5-48c9-b6ec-222e142d4fb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:59.315471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-3ba212d9-619f-4c94-8c78-e25104428ae5 None None] MFA Rules not processed for user `8dbeccfb9a194992959f0b341ef0d4a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:59.318269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-3ba212d9-619f-4c94-8c78-e25104428ae5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:59.448626 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 865/1734] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:59.450592 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8003345f-8d84-4d7d-a6bf-44d361f908ba None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:59.451102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8003345f-8d84-4d7d-a6bf-44d361f908ba None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:59.451607 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8003345f-8d84-4d7d-a6bf-44d361f908ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:59.460777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 870/1735] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Oct 4 14:29:59 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:59.466496 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba212d9-619f-4c94-8c78-e25104428ae5 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:59.472402 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-3ba212d9-619f-4c94-8c78-e25104428ae5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:59.508353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba212d9-619f-4c94-8c78-e25104428ae5 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:59.521083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e46373-5132-48f3-b9eb-96449bfbe018 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:59.521813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e46373-5132-48f3-b9eb-96449bfbe018 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:59.522134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e46373-5132-48f3-b9eb-96449bfbe018 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:59.522482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e46373-5132-48f3-b9eb-96449bfbe018 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:59.525282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-8003345f-8d84-4d7d-a6bf-44d361f908ba None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:59.532428 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8003345f-8d84-4d7d-a6bf-44d361f908ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:59.558777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e46373-5132-48f3-b9eb-96449bfbe018 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:59.565624 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e46373-5132-48f3-b9eb-96449bfbe018 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:59.577777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 871/1736] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:29:59 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:29:59.579333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 866/1737] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:59.580250 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23357f3f-a08a-4813-9586-12c6885dcf95 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:59.580724 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23357f3f-a08a-4813-9586-12c6885dcf95 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:59.581161 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23357f3f-a08a-4813-9586-12c6885dcf95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:59.588629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64825e60-3c72-4c3d-9e0c-766512c2f4e6 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:59.589597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64825e60-3c72-4c3d-9e0c-766512c2f4e6 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:59.590226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64825e60-3c72-4c3d-9e0c-766512c2f4e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:59.610970 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-23357f3f-a08a-4813-9586-12c6885dcf95 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:59.612940 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-23357f3f-a08a-4813-9586-12c6885dcf95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:59.626410 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 872/1738] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:29:59.633589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23357f3f-a08a-4813-9586-12c6885dcf95 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:59.643263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-64825e60-3c72-4c3d-9e0c-766512c2f4e6 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:29:59.644859 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-64825e60-3c72-4c3d-9e0c-766512c2f4e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:29:59.668094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 867/1739] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:29:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:29:59.674155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64825e60-3c72-4c3d-9e0c-766512c2f4e6 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:59.685657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23357f3f-a08a-4813-9586-12c6885dcf95 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:59.696842 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64825e60-3c72-4c3d-9e0c-766512c2f4e6 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:59.703392 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363da13b-ded4-4037-adbe-18eeb6459084 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:59.707238 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363da13b-ded4-4037-adbe-18eeb6459084 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:59.707518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363da13b-ded4-4037-adbe-18eeb6459084 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:59.711967 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363da13b-ded4-4037-adbe-18eeb6459084 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:59.713409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e531155-8d16-4718-bf69-b0a25d037788 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:59.715418 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363da13b-ded4-4037-adbe-18eeb6459084 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:29:59.715890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363da13b-ded4-4037-adbe-18eeb6459084 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:59.716490 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e531155-8d16-4718-bf69-b0a25d037788 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:59.718203 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e531155-8d16-4718-bf69-b0a25d037788 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:59.718718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e531155-8d16-4718-bf69-b0a25d037788 admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:59.719932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e531155-8d16-4718-bf69-b0a25d037788 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:29:59.721642 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e531155-8d16-4718-bf69-b0a25d037788 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:59.722404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363da13b-ded4-4037-adbe-18eeb6459084 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:59.727312 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e531155-8d16-4718-bf69-b0a25d037788 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:59.761864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 868/1740] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:29:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:29:59.765747 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 873/1741] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:29:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:29:59.769625 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e531155-8d16-4718-bf69-b0a25d037788 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:59.772559 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363da13b-ded4-4037-adbe-18eeb6459084 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:59.787548 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e531155-8d16-4718-bf69-b0a25d037788 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:59.797018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363da13b-ded4-4037-adbe-18eeb6459084 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:29:59.803316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d883c3ce-7e75-4554-870c-c692e835e040 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:59.805160 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d883c3ce-7e75-4554-870c-c692e835e040 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:59.805736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d883c3ce-7e75-4554-870c-c692e835e040 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:59.806322 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d883c3ce-7e75-4554-870c-c692e835e040 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:59.808331 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05360610-8033-4b3f-b23a-b985665ce190 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:29:59.808974 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05360610-8033-4b3f-b23a-b985665ce190 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:29:59.809356 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05360610-8033-4b3f-b23a-b985665ce190 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:29:59.809647 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05360610-8033-4b3f-b23a-b985665ce190 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:29:59.818176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05360610-8033-4b3f-b23a-b985665ce190 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:59.821035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d883c3ce-7e75-4554-870c-c692e835e040 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:29:59.821184 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05360610-8033-4b3f-b23a-b985665ce190 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:59.823327 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d883c3ce-7e75-4554-870c-c692e835e040 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:29:59.900781 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 869/1742] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:29:59 2022] POST /identity/v3/projects => generated 389 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:59.902397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d883c3ce-7e75-4554-870c-c692e835e040 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:59.908171 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 874/1743] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:29:59 2022] POST /identity/v3/projects => generated 355 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:29:59.909275 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05360610-8033-4b3f-b23a-b985665ce190 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:29:59.943099 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:00.087788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05360610-8033-4b3f-b23a-b985665ce190 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:00.098482 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d883c3ce-7e75-4554-870c-c692e835e040 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:00.106903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a607e4c5-c38e-4f35-8f1f-4023fdeeb27a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:00.107492 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a607e4c5-c38e-4f35-8f1f-4023fdeeb27a admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:00.107861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a607e4c5-c38e-4f35-8f1f-4023fdeeb27a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:00.108156 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a607e4c5-c38e-4f35-8f1f-4023fdeeb27a admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:00.108873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a607e4c5-c38e-4f35-8f1f-4023fdeeb27a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:00.111107 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a607e4c5-c38e-4f35-8f1f-4023fdeeb27a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:00.115342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b824c944-01de-478c-8b06-d6157ead9315 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:00.117352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b824c944-01de-478c-8b06-d6157ead9315 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:00.117553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b824c944-01de-478c-8b06-d6157ead9315 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:00.117837 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b824c944-01de-478c-8b06-d6157ead9315 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:00.119336 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b824c944-01de-478c-8b06-d6157ead9315 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:00.124273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b824c944-01de-478c-8b06-d6157ead9315 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:00.168740 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 875/1744] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:29:59 2022] POST /identity/v3/users => generated 278 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:00.172079 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a607e4c5-c38e-4f35-8f1f-4023fdeeb27a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:00.177717 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 870/1745] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:29:59 2022] POST /identity/v3/users => generated 295 bytes in 275 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:30:00.182988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b824c944-01de-478c-8b06-d6157ead9315 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:00.201763 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b824c944-01de-478c-8b06-d6157ead9315 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:00.202778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a607e4c5-c38e-4f35-8f1f-4023fdeeb27a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:00.213085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43f8b73-88cd-4058-b9ec-39d9c9e60a79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:00.213722 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43f8b73-88cd-4058-b9ec-39d9c9e60a79 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:00.214012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43f8b73-88cd-4058-b9ec-39d9c9e60a79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:00.214290 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43f8b73-88cd-4058-b9ec-39d9c9e60a79 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:00.215055 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43f8b73-88cd-4058-b9ec-39d9c9e60a79 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:00.216758 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044f81d3-b6a0-46cb-a2f0-86c614fa0ea5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:00.217419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044f81d3-b6a0-46cb-a2f0-86c614fa0ea5 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:00.218121 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044f81d3-b6a0-46cb-a2f0-86c614fa0ea5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:00.218121 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044f81d3-b6a0-46cb-a2f0-86c614fa0ea5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:00.218708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044f81d3-b6a0-46cb-a2f0-86c614fa0ea5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:00.220407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044f81d3-b6a0-46cb-a2f0-86c614fa0ea5 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:00.224095 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43f8b73-88cd-4058-b9ec-39d9c9e60a79 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:00.251442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 871/1746] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:00 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:00.251993 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 876/1747] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:00 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:00.254649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43f8b73-88cd-4058-b9ec-39d9c9e60a79 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:00.256791 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044f81d3-b6a0-46cb-a2f0-86c614fa0ea5 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:00.300701 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044f81d3-b6a0-46cb-a2f0-86c614fa0ea5 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:00.306619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43f8b73-88cd-4058-b9ec-39d9c9e60a79 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:00.328389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5675deb7-c9ff-415e-8586-f17f1599d88b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:00.329017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5675deb7-c9ff-415e-8586-f17f1599d88b admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:00.329296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5675deb7-c9ff-415e-8586-f17f1599d88b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:00.329598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5675deb7-c9ff-415e-8586-f17f1599d88b admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:00.330198 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5675deb7-c9ff-415e-8586-f17f1599d88b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:00.331807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5675deb7-c9ff-415e-8586-f17f1599d88b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:00.347831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f716bd1-08ac-4d03-ad4a-01fc33fbfdb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:00.348515 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f716bd1-08ac-4d03-ad4a-01fc33fbfdb6 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:00.348862 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f716bd1-08ac-4d03-ad4a-01fc33fbfdb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:00.349152 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f716bd1-08ac-4d03-ad4a-01fc33fbfdb6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:00.349853 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f716bd1-08ac-4d03-ad4a-01fc33fbfdb6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:00.366144 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f716bd1-08ac-4d03-ad4a-01fc33fbfdb6 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:00.400578 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 877/1748] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:00 2022] GET /identity/v3/roles => generated 1420 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:00.401827 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5675deb7-c9ff-415e-8586-f17f1599d88b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:00.414215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5675deb7-c9ff-415e-8586-f17f1599d88b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:00.427684 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a543647-e70e-4b67-81e1-aa242caa8e1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:00.428487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a543647-e70e-4b67-81e1-aa242caa8e1c admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:00.431182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a543647-e70e-4b67-81e1-aa242caa8e1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:00.432401 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a543647-e70e-4b67-81e1-aa242caa8e1c admin admin] PATH_INFO: `/v3/projects/3ce87fdc82234e459b065cbf584eb5f0/users/388d44d8a2ac4efb96407b5a184b02e7/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:00.453675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 872/1749] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:00 2022] GET /identity/v3/roles => generated 1420 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:00.454926 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f716bd1-08ac-4d03-ad4a-01fc33fbfdb6 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:00.468358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a543647-e70e-4b67-81e1-aa242caa8e1c admin admin] RBAC: Authorizing `identity:create_grant(project_id=3ce87fdc82234e459b065cbf584eb5f0, user_id=388d44d8a2ac4efb96407b5a184b02e7, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:00.473447 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a543647-e70e-4b67-81e1-aa242caa8e1c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:00.491879 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f716bd1-08ac-4d03-ad4a-01fc33fbfdb6 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:00.539979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:00.541335 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 878/1750] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:00 2022] PUT /identity/v3/projects/3ce87fdc82234e459b065cbf584eb5f0/users/388d44d8a2ac4efb96407b5a184b02e7/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:00.544156 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1456f2df-b435-4327-9b9d-cdd891816786 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:00.544863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1456f2df-b435-4327-9b9d-cdd891816786 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:00.545392 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1456f2df-b435-4327-9b9d-cdd891816786 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:00.650884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460928e1-d61d-4eb6-a0af-833352ecaacf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:00.656002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460928e1-d61d-4eb6-a0af-833352ecaacf admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:00.656282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460928e1-d61d-4eb6-a0af-833352ecaacf admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:00.658254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460928e1-d61d-4eb6-a0af-833352ecaacf admin admin] PATH_INFO: `/v3/projects/a5acd1c20dbb414588bf9fa0d89cae3a/users/8369731f29994ef18714381ffea041ea/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:00.713501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460928e1-d61d-4eb6-a0af-833352ecaacf admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5acd1c20dbb414588bf9fa0d89cae3a, user_id=8369731f29994ef18714381ffea041ea, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:00.716746 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460928e1-d61d-4eb6-a0af-833352ecaacf admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:00.753637 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 873/1751] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:00 2022] PUT /identity/v3/projects/a5acd1c20dbb414588bf9fa0d89cae3a/users/8369731f29994ef18714381ffea041ea/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:00.755843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e539a4-8ed2-4a71-aaa6-958d8049ddf3 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:00.756273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e539a4-8ed2-4a71-aaa6-958d8049ddf3 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:00.757065 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e539a4-8ed2-4a71-aaa6-958d8049ddf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:00.809890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-1456f2df-b435-4327-9b9d-cdd891816786 None None] MFA Rules not processed for user `34afa4aac10c4a018ca05a0ba34c940a`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:00.815585 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1456f2df-b435-4327-9b9d-cdd891816786 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:00.858095 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-a2e539a4-8ed2-4a71-aaa6-958d8049ddf3 None None] MFA Rules not processed for user `388d44d8a2ac4efb96407b5a184b02e7`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:00.861555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a2e539a4-8ed2-4a71-aaa6-958d8049ddf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:01.015548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 879/1752] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Oct 4 14:30:00 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 464 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:01.022416 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1456f2df-b435-4327-9b9d-cdd891816786 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:01.026345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1456f2df-b435-4327-9b9d-cdd891816786 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:01.088265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 874/1753] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:00 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 334 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:01.090430 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a249aeb-af07-4ea5-870d-84b0961300d5 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:01.090925 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a249aeb-af07-4ea5-870d-84b0961300d5 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:01.096071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a249aeb-af07-4ea5-870d-84b0961300d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:01.177880 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:01.224630 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-3a249aeb-af07-4ea5-870d-84b0961300d5 None None] MFA Rules not processed for user `8369731f29994ef18714381ffea041ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:01.228587 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-3a249aeb-af07-4ea5-870d-84b0961300d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:01.292751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1456f2df-b435-4327-9b9d-cdd891816786 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:01.302130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb40fbfb-dc9a-4711-bbb0-6fd16e9c7ee1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:01.302923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb40fbfb-dc9a-4711-bbb0-6fd16e9c7ee1 service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:01.303203 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb40fbfb-dc9a-4711-bbb0-6fd16e9c7ee1 service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:01.304020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb40fbfb-dc9a-4711-bbb0-6fd16e9c7ee1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:01.307319 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-cb40fbfb-dc9a-4711-bbb0-6fd16e9c7ee1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:01.597286 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 875/1754] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:01 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 508 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:01.598290 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a249aeb-af07-4ea5-870d-84b0961300d5 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:01.660200 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb40fbfb-dc9a-4711-bbb0-6fd16e9c7ee1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:01.666002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb40fbfb-dc9a-4711-bbb0-6fd16e9c7ee1 service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:01.705495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 880/1755] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:01 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 689 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:01.706764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb40fbfb-dc9a-4711-bbb0-6fd16e9c7ee1 service nova] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:01.711781 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-cb40fbfb-dc9a-4711-bbb0-6fd16e9c7ee1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:01.747468 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:02.023307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a249aeb-af07-4ea5-870d-84b0961300d5 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:02.053916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cd5c2f-9a60-4aca-928c-e62e5118af0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:02.054756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cd5c2f-9a60-4aca-928c-e62e5118af0a admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:02.059381 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cd5c2f-9a60-4aca-928c-e62e5118af0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:02.059676 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cd5c2f-9a60-4aca-928c-e62e5118af0a admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:02.060405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cd5c2f-9a60-4aca-928c-e62e5118af0a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:02.074617 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cd5c2f-9a60-4aca-928c-e62e5118af0a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:02.210681 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 876/1756] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:30:01 2022] POST /identity/v3/projects => generated 387 bytes in 612 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:02.215142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cd5c2f-9a60-4aca-928c-e62e5118af0a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:02.379231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb40fbfb-dc9a-4711-bbb0-6fd16e9c7ee1 service nova] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:02.598377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cd5c2f-9a60-4aca-928c-e62e5118af0a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:02.629150 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa1a662-b322-421b-b784-00a63042572d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:02.630646 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa1a662-b322-421b-b784-00a63042572d admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:02.634047 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa1a662-b322-421b-b784-00a63042572d admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:02.635071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa1a662-b322-421b-b784-00a63042572d admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:02.637303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa1a662-b322-421b-b784-00a63042572d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:02.643730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa1a662-b322-421b-b784-00a63042572d admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:02.660071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:02.762454 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87ad424-cf0a-4925-b5e8-7fdbcbe8dea6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5980e353164c4873bcdd2af847e0a1f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:02.768800 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87ad424-cf0a-4925-b5e8-7fdbcbe8dea6 service swift] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:02.768800 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87ad424-cf0a-4925-b5e8-7fdbcbe8dea6 service swift] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:02.769220 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87ad424-cf0a-4925-b5e8-7fdbcbe8dea6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:02.897332 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 877/1757] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:30:02 2022] POST /identity/v3/projects => generated 357 bytes in 687 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:02.898257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa1a662-b322-421b-b784-00a63042572d admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:02.970924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87ad424-cf0a-4925-b5e8-7fdbcbe8dea6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:02.973214 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87ad424-cf0a-4925-b5e8-7fdbcbe8dea6 service swift] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:03.006923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 881/1758] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Oct 4 14:30:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 1301 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 14:30:03.008009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87ad424-cf0a-4925-b5e8-7fdbcbe8dea6 service swift] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:03.063350 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87ad424-cf0a-4925-b5e8-7fdbcbe8dea6 service swift] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:03.161211 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa1a662-b322-421b-b784-00a63042572d admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:03.177286 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c3d559-9432-4843-b455-23967cf17e04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:03.178730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c3d559-9432-4843-b455-23967cf17e04 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:03.179332 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c3d559-9432-4843-b455-23967cf17e04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:03.179831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c3d559-9432-4843-b455-23967cf17e04 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:03.181792 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c3d559-9432-4843-b455-23967cf17e04 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:03.219376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c3d559-9432-4843-b455-23967cf17e04 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:03.268761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e22b26-da8e-4d2c-b316-10bfe3b35ae0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:03.269395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e22b26-da8e-4d2c-b316-10bfe3b35ae0 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:03.269670 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e22b26-da8e-4d2c-b316-10bfe3b35ae0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:03.269940 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e22b26-da8e-4d2c-b316-10bfe3b35ae0 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:03.271379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e22b26-da8e-4d2c-b316-10bfe3b35ae0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:03.275114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e22b26-da8e-4d2c-b316-10bfe3b35ae0 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:03.295579 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 878/1759] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:30:02 2022] POST /identity/v3/users => generated 300 bytes in 397 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:03.299493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132d9cf4-dd27-4810-b7a9-abf3035279c6 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:03.299789 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132d9cf4-dd27-4810-b7a9-abf3035279c6 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:03.300073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132d9cf4-dd27-4810-b7a9-abf3035279c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:03.357473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 882/1760] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:30:03 2022] POST /identity/v3/users => generated 285 bytes in 348 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:03.360468 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e22b26-da8e-4d2c-b316-10bfe3b35ae0 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:03.377127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-132d9cf4-dd27-4810-b7a9-abf3035279c6 None None] MFA Rules not processed for user `c5b38375d6ff4d0db0f3f7bc89ca8798`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:03.380679 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-132d9cf4-dd27-4810-b7a9-abf3035279c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:03.401461 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e22b26-da8e-4d2c-b316-10bfe3b35ae0 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:03.415492 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6563673d-1555-4ba7-a414-873d7aee8b2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:03.416967 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6563673d-1555-4ba7-a414-873d7aee8b2a admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:03.417408 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6563673d-1555-4ba7-a414-873d7aee8b2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:03.417921 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6563673d-1555-4ba7-a414-873d7aee8b2a admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:03.418900 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6563673d-1555-4ba7-a414-873d7aee8b2a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:03.421189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6563673d-1555-4ba7-a414-873d7aee8b2a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:03.436459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 883/1761] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:03 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:03.437579 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6563673d-1555-4ba7-a414-873d7aee8b2a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:03.441380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:03.465737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6563673d-1555-4ba7-a414-873d7aee8b2a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:03.508242 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574ed7a6-5b46-40a6-9586-c7dcd0bc76c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:03.515704 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574ed7a6-5b46-40a6-9586-c7dcd0bc76c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:03.519581 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574ed7a6-5b46-40a6-9586-c7dcd0bc76c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:03.523624 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574ed7a6-5b46-40a6-9586-c7dcd0bc76c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:03.525223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574ed7a6-5b46-40a6-9586-c7dcd0bc76c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:03.540226 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574ed7a6-5b46-40a6-9586-c7dcd0bc76c6 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:03.589986 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 879/1762] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:03 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:03.590614 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 884/1763] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:03 2022] GET /identity/v3/roles => generated 1420 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:03.591703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574ed7a6-5b46-40a6-9586-c7dcd0bc76c6 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:03.596743 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132d9cf4-dd27-4810-b7a9-abf3035279c6 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:03.633888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574ed7a6-5b46-40a6-9586-c7dcd0bc76c6 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:03.637599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132d9cf4-dd27-4810-b7a9-abf3035279c6 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:03.649271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed907bea-7002-47ef-8009-8c5dc557a34c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:03.651387 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed907bea-7002-47ef-8009-8c5dc557a34c admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:03.652656 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed907bea-7002-47ef-8009-8c5dc557a34c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:03.654040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed907bea-7002-47ef-8009-8c5dc557a34c admin admin] PATH_INFO: `/v3/projects/66e6a072abf54472b42c30881306fa58/users/4f81900a54f84c1087dd66257c7d8de8/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:03.655424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be843381-a53a-4ed9-baf9-686ebd48e048 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:03.656804 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be843381-a53a-4ed9-baf9-686ebd48e048 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:03.660408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be843381-a53a-4ed9-baf9-686ebd48e048 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:03.662414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be843381-a53a-4ed9-baf9-686ebd48e048 admin admin] PATH_INFO: `/v3/projects/de2286acf89b45a89d72949ea5783335/users/6e9bf40d49a94a12a4b9ad2dc243f1b2/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:03.725739 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed907bea-7002-47ef-8009-8c5dc557a34c admin admin] RBAC: Authorizing `identity:create_grant(project_id=66e6a072abf54472b42c30881306fa58, user_id=4f81900a54f84c1087dd66257c7d8de8, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:03.728592 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be843381-a53a-4ed9-baf9-686ebd48e048 admin admin] RBAC: Authorizing `identity:create_grant(project_id=de2286acf89b45a89d72949ea5783335, user_id=6e9bf40d49a94a12a4b9ad2dc243f1b2, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:03.736101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be843381-a53a-4ed9-baf9-686ebd48e048 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:03.740154 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed907bea-7002-47ef-8009-8c5dc557a34c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:03.803769 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 880/1764] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:03 2022] PUT /identity/v3/projects/de2286acf89b45a89d72949ea5783335/users/6e9bf40d49a94a12a4b9ad2dc243f1b2/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:03.803947 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 885/1765] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:03 2022] PUT /identity/v3/projects/66e6a072abf54472b42c30881306fa58/users/4f81900a54f84c1087dd66257c7d8de8/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:03.805933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be843381-a53a-4ed9-baf9-686ebd48e048 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:03.807406 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed907bea-7002-47ef-8009-8c5dc557a34c admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:03.908635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:04.048098 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed907bea-7002-47ef-8009-8c5dc557a34c admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:04.071079 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b904a1-b2b7-4182-864a-acffe49866d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:04.076370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b904a1-b2b7-4182-864a-acffe49866d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:04.077253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b904a1-b2b7-4182-864a-acffe49866d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:04.077813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b904a1-b2b7-4182-864a-acffe49866d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:04.078838 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b904a1-b2b7-4182-864a-acffe49866d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:04.081068 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b904a1-b2b7-4182-864a-acffe49866d3 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:04.128010 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be843381-a53a-4ed9-baf9-686ebd48e048 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:04.138946 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 886/1766] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:03 2022] GET /identity/v3/roles => generated 1420 bytes in 329 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:04.141165 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b904a1-b2b7-4182-864a-acffe49866d3 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:04.158049 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b904a1-b2b7-4182-864a-acffe49866d3 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:04.164059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e014d44-b270-4ed2-a3c3-9196aa1d61d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:04.165400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e014d44-b270-4ed2-a3c3-9196aa1d61d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:04.165618 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0058fd-c6d5-4310-919a-ec8ed58cee9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:04.167171 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0058fd-c6d5-4310-919a-ec8ed58cee9e admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:04.168014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e014d44-b270-4ed2-a3c3-9196aa1d61d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:04.168207 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0058fd-c6d5-4310-919a-ec8ed58cee9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:04.168998 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0058fd-c6d5-4310-919a-ec8ed58cee9e admin admin] PATH_INFO: `/v3/projects/de2286acf89b45a89d72949ea5783335/users/6e9bf40d49a94a12a4b9ad2dc243f1b2/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:04.170274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e014d44-b270-4ed2-a3c3-9196aa1d61d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:04.171023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e014d44-b270-4ed2-a3c3-9196aa1d61d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:04.171741 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0058fd-c6d5-4310-919a-ec8ed58cee9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=de2286acf89b45a89d72949ea5783335, user_id=6e9bf40d49a94a12a4b9ad2dc243f1b2, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:04.175213 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0058fd-c6d5-4310-919a-ec8ed58cee9e admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:04.180482 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e014d44-b270-4ed2-a3c3-9196aa1d61d8 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:04.193384 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 887/1767] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:04 2022] PUT /identity/v3/projects/de2286acf89b45a89d72949ea5783335/users/6e9bf40d49a94a12a4b9ad2dc243f1b2/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:04.211470 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63ae9e6-274b-4a8e-84b6-4600059a60ea None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:04.214839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63ae9e6-274b-4a8e-84b6-4600059a60ea None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:04.220819 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63ae9e6-274b-4a8e-84b6-4600059a60ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:04.229747 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 881/1768] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:03 2022] GET /identity/v3/roles => generated 1420 bytes in 424 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:04.237074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e014d44-b270-4ed2-a3c3-9196aa1d61d8 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:04.290063 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-e63ae9e6-274b-4a8e-84b6-4600059a60ea None None] MFA Rules not processed for user `6e9bf40d49a94a12a4b9ad2dc243f1b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:04.293570 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-e63ae9e6-274b-4a8e-84b6-4600059a60ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:04.298232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e014d44-b270-4ed2-a3c3-9196aa1d61d8 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:04.604643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f5eef3-de82-4dac-bf38-21b2a866e041 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:04.605539 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f5eef3-de82-4dac-bf38-21b2a866e041 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:04.605839 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f5eef3-de82-4dac-bf38-21b2a866e041 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:04.612914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f5eef3-de82-4dac-bf38-21b2a866e041 admin admin] PATH_INFO: `/v3/projects/66e6a072abf54472b42c30881306fa58/users/4f81900a54f84c1087dd66257c7d8de8/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:04.627185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f5eef3-de82-4dac-bf38-21b2a866e041 admin admin] RBAC: Authorizing `identity:create_grant(project_id=66e6a072abf54472b42c30881306fa58, user_id=4f81900a54f84c1087dd66257c7d8de8, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:04.627986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 888/1769] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:04 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 416 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:04.629991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f5eef3-de82-4dac-bf38-21b2a866e041 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:04.636776 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63ae9e6-274b-4a8e-84b6-4600059a60ea None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:04.639730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-e63ae9e6-274b-4a8e-84b6-4600059a60ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:04.762162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 882/1770] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:04 2022] PUT /identity/v3/projects/66e6a072abf54472b42c30881306fa58/users/4f81900a54f84c1087dd66257c7d8de8/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 510 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:04.762162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c918f319-76ce-45ec-a618-e7d3f397e6e4 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:04.762162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c918f319-76ce-45ec-a618-e7d3f397e6e4 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:04.762162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c918f319-76ce-45ec-a618-e7d3f397e6e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:04.885515 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63ae9e6-274b-4a8e-84b6-4600059a60ea None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:04.966140 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-c918f319-76ce-45ec-a618-e7d3f397e6e4 None None] MFA Rules not processed for user `4f81900a54f84c1087dd66257c7d8de8`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:04.974745 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c918f319-76ce-45ec-a618-e7d3f397e6e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:05.172875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:05.203947 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f42162-b1dc-4058-8686-14550331fd38 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:05.204224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f42162-b1dc-4058-8686-14550331fd38 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:05.206187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f42162-b1dc-4058-8686-14550331fd38 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:05.207769 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f42162-b1dc-4058-8686-14550331fd38 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:05.251824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:05.402096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 883/1771] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:04 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 644 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:05.428588 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c918f319-76ce-45ec-a618-e7d3f397e6e4 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:05.432131 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c918f319-76ce-45ec-a618-e7d3f397e6e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:05.463788 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c918f319-76ce-45ec-a618-e7d3f397e6e4 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:05.489312 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e221d960-c28e-4bef-802f-32fd9745117b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:05.489902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e221d960-c28e-4bef-802f-32fd9745117b service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:05.490209 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e221d960-c28e-4bef-802f-32fd9745117b service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:05.490472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e221d960-c28e-4bef-802f-32fd9745117b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:05.537061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e221d960-c28e-4bef-802f-32fd9745117b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:05.543657 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e221d960-c28e-4bef-802f-32fd9745117b service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:05.582401 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f42162-b1dc-4058-8686-14550331fd38 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:05.586164 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f42162-b1dc-4058-8686-14550331fd38 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:05.587656 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 884/1772] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:05 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:30:05.627423 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 889/1773] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:04 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 992 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:30:05.950801 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e221d960-c28e-4bef-802f-32fd9745117b service cinder] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:05.988770 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e221d960-c28e-4bef-802f-32fd9745117b service cinder] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:06.003344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f18515-3aac-4d47-86b1-fd087832f201 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:06.005971 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f18515-3aac-4d47-86b1-fd087832f201 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:06.006959 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f18515-3aac-4d47-86b1-fd087832f201 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:06.007434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f18515-3aac-4d47-86b1-fd087832f201 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:06.231545 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f18515-3aac-4d47-86b1-fd087832f201 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:06.233144 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f18515-3aac-4d47-86b1-fd087832f201 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:06.265735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 885/1774] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:05 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:06.279724 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9201aa1b-2f9c-4f11-b13d-3ba2be7df222 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:06.280278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9201aa1b-2f9c-4f11-b13d-3ba2be7df222 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:06.280278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9201aa1b-2f9c-4f11-b13d-3ba2be7df222 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:06.360222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-9201aa1b-2f9c-4f11-b13d-3ba2be7df222 None None] MFA Rules not processed for user `8dbeccfb9a194992959f0b341ef0d4a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:06.364562 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9201aa1b-2f9c-4f11-b13d-3ba2be7df222 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:06.703075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 890/1775] 10.222.0.17 () {58 vars in 1015 bytes} [Tue Oct 4 14:30:06 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 425 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:30:06.954215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f18515-3aac-4d47-86b1-fd087832f201 service cinder] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:06.985520 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f18515-3aac-4d47-86b1-fd087832f201 service cinder] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:07.000931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8438a4b6-3e39-4c03-881b-070fdcfc5b8a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:07.002161 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8438a4b6-3e39-4c03-881b-070fdcfc5b8a service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:07.002639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8438a4b6-3e39-4c03-881b-070fdcfc5b8a service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:07.003105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8438a4b6-3e39-4c03-881b-070fdcfc5b8a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:07.029750 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8438a4b6-3e39-4c03-881b-070fdcfc5b8a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:07.031701 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8438a4b6-3e39-4c03-881b-070fdcfc5b8a service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:07.055970 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 886/1776] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:06 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:07.143479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9201aa1b-2f9c-4f11-b13d-3ba2be7df222 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:07.658024 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9201aa1b-2f9c-4f11-b13d-3ba2be7df222 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:07.675703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fb77e1-826f-4269-82d0-bf29a000a499 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:07.678114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fb77e1-826f-4269-82d0-bf29a000a499 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:07.679239 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fb77e1-826f-4269-82d0-bf29a000a499 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:07.681113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fb77e1-826f-4269-82d0-bf29a000a499 admin admin] PATH_INFO: `/v3/users/388d44d8a2ac4efb96407b5a184b02e7` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:07.688011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fb77e1-826f-4269-82d0-bf29a000a499 admin admin] RBAC: Authorizing `identity:delete_user(user_id=388d44d8a2ac4efb96407b5a184b02e7)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:07.692000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fb77e1-826f-4269-82d0-bf29a000a499 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:07.900115 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-34fb77e1-826f-4269-82d0-bf29a000a499 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '388d44d8a2ac4efb96407b5a184b02e7'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:07.939215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-34fb77e1-826f-4269-82d0-bf29a000a499 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '388d44d8a2ac4efb96407b5a184b02e7'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:07.963699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:07.970288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-34fb77e1-826f-4269-82d0-bf29a000a499 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '388d44d8a2ac4efb96407b5a184b02e7'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:08.011744 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-34fb77e1-826f-4269-82d0-bf29a000a499 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '388d44d8a2ac4efb96407b5a184b02e7'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:08.014291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-34fb77e1-826f-4269-82d0-bf29a000a499 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '388d44d8a2ac4efb96407b5a184b02e7'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:08.038455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 891/1777] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:30:07 2022] DELETE /identity/v3/users/388d44d8a2ac4efb96407b5a184b02e7 => generated 0 bytes in 891 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:08.039808 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b92bb-b427-47e2-bcc5-4810c0eb25d5 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:08.040724 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b92bb-b427-47e2-bcc5-4810c0eb25d5 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:08.041166 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b92bb-b427-47e2-bcc5-4810c0eb25d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:08.102268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-555b92bb-b427-47e2-bcc5-4810c0eb25d5 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:08.105259 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-555b92bb-b427-47e2-bcc5-4810c0eb25d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:08.372681 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 887/1778] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 333 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:08.498014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fb77e1-826f-4269-82d0-bf29a000a499 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:08.501005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-34fb77e1-826f-4269-82d0-bf29a000a499 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:08.653133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fb77e1-826f-4269-82d0-bf29a000a499 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:08.663710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f9ebba-2e19-44d8-a6b5-d8da6a865f52 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:08.665504 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f9ebba-2e19-44d8-a6b5-d8da6a865f52 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:08.666759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f9ebba-2e19-44d8-a6b5-d8da6a865f52 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:08.667251 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f9ebba-2e19-44d8-a6b5-d8da6a865f52 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:08.688549 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f9ebba-2e19-44d8-a6b5-d8da6a865f52 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:08.690347 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f9ebba-2e19-44d8-a6b5-d8da6a865f52 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:08.708549 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 892/1779] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:14.788442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555b92bb-b427-47e2-bcc5-4810c0eb25d5 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:14.789988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-555b92bb-b427-47e2-bcc5-4810c0eb25d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:14.814852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555b92bb-b427-47e2-bcc5-4810c0eb25d5 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:14.844389 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db01f435-332e-45c8-87f8-c69aebbf7a72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:14.845202 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db01f435-332e-45c8-87f8-c69aebbf7a72 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:14.845547 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db01f435-332e-45c8-87f8-c69aebbf7a72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:14.845896 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db01f435-332e-45c8-87f8-c69aebbf7a72 admin admin] PATH_INFO: `/v3/projects/3ce87fdc82234e459b065cbf584eb5f0` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:14.853294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db01f435-332e-45c8-87f8-c69aebbf7a72 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3ce87fdc82234e459b065cbf584eb5f0)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:14.864437 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db01f435-332e-45c8-87f8-c69aebbf7a72 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:14.968148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:15.046872 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-db01f435-332e-45c8-87f8-c69aebbf7a72 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3ce87fdc82234e459b065cbf584eb5f0', 'request_id': 'req-db01f435-332e-45c8-87f8-c69aebbf7a72'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:15.101772 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-db01f435-332e-45c8-87f8-c69aebbf7a72 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3ce87fdc82234e459b065cbf584eb5f0', 'request_id': 'req-db01f435-332e-45c8-87f8-c69aebbf7a72'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:15.149934 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-db01f435-332e-45c8-87f8-c69aebbf7a72 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3ce87fdc82234e459b065cbf584eb5f0', 'request_id': 'req-db01f435-332e-45c8-87f8-c69aebbf7a72'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:15.185199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-db01f435-332e-45c8-87f8-c69aebbf7a72 admin admin] The token cache is being invalidate because project 3ce87fdc82234e459b065cbf584eb5f0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:30:15.185473 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-db01f435-332e-45c8-87f8-c69aebbf7a72 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:15.191484 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-db01f435-332e-45c8-87f8-c69aebbf7a72 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:15.198299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 888/1780] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:30:14 2022] DELETE /identity/v3/projects/3ce87fdc82234e459b065cbf584eb5f0 => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:15.202046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f9ebba-2e19-44d8-a6b5-d8da6a865f52 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:15.232411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f9ebba-2e19-44d8-a6b5-d8da6a865f52 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:15.551443 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a365db-2741-4562-a244-70b6ae1cd78d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:15.552179 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a365db-2741-4562-a244-70b6ae1cd78d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:15.552493 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a365db-2741-4562-a244-70b6ae1cd78d admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:15.553346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a365db-2741-4562-a244-70b6ae1cd78d admin admin] PATH_INFO: `/v3/users/6e9bf40d49a94a12a4b9ad2dc243f1b2` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:15.561304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a365db-2741-4562-a244-70b6ae1cd78d admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e9bf40d49a94a12a4b9ad2dc243f1b2)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:15.565042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a365db-2741-4562-a244-70b6ae1cd78d admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:15.784065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-88a365db-2741-4562-a244-70b6ae1cd78d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e9bf40d49a94a12a4b9ad2dc243f1b2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:15.827701 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-88a365db-2741-4562-a244-70b6ae1cd78d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e9bf40d49a94a12a4b9ad2dc243f1b2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:15.845192 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-88a365db-2741-4562-a244-70b6ae1cd78d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e9bf40d49a94a12a4b9ad2dc243f1b2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:15.925642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-88a365db-2741-4562-a244-70b6ae1cd78d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e9bf40d49a94a12a4b9ad2dc243f1b2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:15.926675 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-88a365db-2741-4562-a244-70b6ae1cd78d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e9bf40d49a94a12a4b9ad2dc243f1b2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:15.934164 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:15.955122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 893/1781] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:30:15 2022] DELETE /identity/v3/users/6e9bf40d49a94a12a4b9ad2dc243f1b2 => generated 0 bytes in 749 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:16.444249 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9f43eb-6680-43e8-b8eb-e7e2d09f13a4 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:16.444951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9f43eb-6680-43e8-b8eb-e7e2d09f13a4 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:16.445065 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9f43eb-6680-43e8-b8eb-e7e2d09f13a4 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:16.446545 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 889/1782] 10.222.0.64 () {54 vars in 921 bytes} [Tue Oct 4 14:30:16 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:30:16.610167 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a4644e-c988-4099-a795-b5438dcf8848 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:16.611221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a4644e-c988-4099-a795-b5438dcf8848 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:16.611808 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a4644e-c988-4099-a795-b5438dcf8848 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:16.721216 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-76a4644e-c988-4099-a795-b5438dcf8848 None None] MFA Rules not processed for user `03081d7d581745c88c12e9d089fcb2d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:16.723684 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-76a4644e-c988-4099-a795-b5438dcf8848 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:17.123312 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 894/1783] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Oct 4 14:30:16 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 514 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:17.942240 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac1fa61-31ee-4e8c-802d-baba8beac05d None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:17.942744 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac1fa61-31ee-4e8c-802d-baba8beac05d None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:17.942744 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac1fa61-31ee-4e8c-802d-baba8beac05d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:17.994525 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-eac1fa61-31ee-4e8c-802d-baba8beac05d None None] MFA Rules not processed for user `fc02bb4b8a32400faef8743d2418a648`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:17.996552 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-eac1fa61-31ee-4e8c-802d-baba8beac05d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:18.184947 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 890/1784] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:17 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:18.410460 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a4644e-c988-4099-a795-b5438dcf8848 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:18.414138 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-76a4644e-c988-4099-a795-b5438dcf8848 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:18.436809 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a4644e-c988-4099-a795-b5438dcf8848 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:18.444746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cf5a7d-4f53-432b-b8e6-c6d617b28cec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:18.445767 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cf5a7d-4f53-432b-b8e6-c6d617b28cec service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:18.446295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cf5a7d-4f53-432b-b8e6-c6d617b28cec service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:18.446850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cf5a7d-4f53-432b-b8e6-c6d617b28cec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:18.477837 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cf5a7d-4f53-432b-b8e6-c6d617b28cec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:18.480783 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cf5a7d-4f53-432b-b8e6-c6d617b28cec service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:18.518222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 895/1785] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:19.121351 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb896a9c-ac52-4f6f-961b-ba7108829a30 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:19.121351 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb896a9c-ac52-4f6f-961b-ba7108829a30 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:19.123509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb896a9c-ac52-4f6f-961b-ba7108829a30 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:19.124581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 891/1786] 10.222.0.64 () {54 vars in 921 bytes} [Tue Oct 4 14:30:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:30:19.155410 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16431876-e266-4161-bc43-345f52df452f None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:19.155696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16431876-e266-4161-bc43-345f52df452f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:19.155968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16431876-e266-4161-bc43-345f52df452f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:19.195866 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-16431876-e266-4161-bc43-345f52df452f None None] MFA Rules not processed for user `03081d7d581745c88c12e9d089fcb2d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:19.197125 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-16431876-e266-4161-bc43-345f52df452f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:19.233161 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 896/1787] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Oct 4 14:30:19 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:19.476875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb896a9c-ac52-4f6f-961b-ba7108829a30 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:19.481613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-cb896a9c-ac52-4f6f-961b-ba7108829a30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:19.880545 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb896a9c-ac52-4f6f-961b-ba7108829a30 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:19.928449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725eb257-f0bd-4347-9645-fd563c0a60b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:19.936800 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725eb257-f0bd-4347-9645-fd563c0a60b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:19.937318 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725eb257-f0bd-4347-9645-fd563c0a60b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:19.938034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725eb257-f0bd-4347-9645-fd563c0a60b8 admin admin] PATH_INFO: `/v3/projects/de2286acf89b45a89d72949ea5783335` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:19.947982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725eb257-f0bd-4347-9645-fd563c0a60b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=de2286acf89b45a89d72949ea5783335)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:19.956836 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725eb257-f0bd-4347-9645-fd563c0a60b8 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:20.053854 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16431876-e266-4161-bc43-345f52df452f None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:20.071799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16431876-e266-4161-bc43-345f52df452f None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:20.096472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-725eb257-f0bd-4347-9645-fd563c0a60b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'de2286acf89b45a89d72949ea5783335', 'request_id': 'req-725eb257-f0bd-4347-9645-fd563c0a60b8'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:20.119001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-725eb257-f0bd-4347-9645-fd563c0a60b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'de2286acf89b45a89d72949ea5783335', 'request_id': 'req-725eb257-f0bd-4347-9645-fd563c0a60b8'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:20.141069 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:20.147596 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-725eb257-f0bd-4347-9645-fd563c0a60b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'de2286acf89b45a89d72949ea5783335', 'request_id': 'req-725eb257-f0bd-4347-9645-fd563c0a60b8'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:20.184298 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-725eb257-f0bd-4347-9645-fd563c0a60b8 admin admin] The token cache is being invalidate because project de2286acf89b45a89d72949ea5783335 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:30:20.184971 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-725eb257-f0bd-4347-9645-fd563c0a60b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:20.186124 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-725eb257-f0bd-4347-9645-fd563c0a60b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:20.191105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 892/1788] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:30:19 2022] DELETE /identity/v3/projects/de2286acf89b45a89d72949ea5783335 => generated 0 bytes in 715 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:20.199326 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5c4ad6-79dd-4221-a394-afae22f5c35c None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:20.199764 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5c4ad6-79dd-4221-a394-afae22f5c35c None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:20.200044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5c4ad6-79dd-4221-a394-afae22f5c35c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:20.268482 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-ce5c4ad6-79dd-4221-a394-afae22f5c35c None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:20.269830 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ce5c4ad6-79dd-4221-a394-afae22f5c35c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:20.273239 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16aee258-fc29-431d-a8ce-14dd23e6090f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:20.274101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16aee258-fc29-431d-a8ce-14dd23e6090f service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:20.274527 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16aee258-fc29-431d-a8ce-14dd23e6090f service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:20.274934 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16aee258-fc29-431d-a8ce-14dd23e6090f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:20.276840 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-16aee258-fc29-431d-a8ce-14dd23e6090f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:20.306431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16aee258-fc29-431d-a8ce-14dd23e6090f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:20.309431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16aee258-fc29-431d-a8ce-14dd23e6090f service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:20.333824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 897/1789] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:20 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 281 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:30:20.387381 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 893/1790] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:30:20.403709 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f6759c-73cc-4ddd-8692-b262f704f154 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:20.404550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f6759c-73cc-4ddd-8692-b262f704f154 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:20.408502 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f6759c-73cc-4ddd-8692-b262f704f154 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:20.482479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-50f6759c-73cc-4ddd-8692-b262f704f154 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:20.487018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-50f6759c-73cc-4ddd-8692-b262f704f154 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:20.529636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 898/1791] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:20.531436 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5c4ad6-79dd-4221-a394-afae22f5c35c None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:20.546301 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5c4ad6-79dd-4221-a394-afae22f5c35c None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:20.557088 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58feb17b-c760-4a2a-ba73-f5251c16afa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:20.557716 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58feb17b-c760-4a2a-ba73-f5251c16afa2 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:20.561734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58feb17b-c760-4a2a-ba73-f5251c16afa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:20.561734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58feb17b-c760-4a2a-ba73-f5251c16afa2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:20.562375 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58feb17b-c760-4a2a-ba73-f5251c16afa2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:30:20.562685 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58feb17b-c760-4a2a-ba73-f5251c16afa2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:20.564598 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58feb17b-c760-4a2a-ba73-f5251c16afa2 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:20.605342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 894/1792] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:30:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:30:20.609200 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f6759c-73cc-4ddd-8692-b262f704f154 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:20.635882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f6759c-73cc-4ddd-8692-b262f704f154 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:20.649599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c19cc6b-dd40-40c9-a7da-d218d5a06b2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:20.650534 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c19cc6b-dd40-40c9-a7da-d218d5a06b2b admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:20.651836 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c19cc6b-dd40-40c9-a7da-d218d5a06b2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:20.652351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c19cc6b-dd40-40c9-a7da-d218d5a06b2b admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:20.655950 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c19cc6b-dd40-40c9-a7da-d218d5a06b2b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:20.659048 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c19cc6b-dd40-40c9-a7da-d218d5a06b2b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:20.731372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 899/1793] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:30:20 2022] POST /identity/v3/projects => generated 369 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:20.734946 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58feb17b-c760-4a2a-ba73-f5251c16afa2 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:20.749907 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58feb17b-c760-4a2a-ba73-f5251c16afa2 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:20.864873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc86852-627c-45f6-998c-19dece304a2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:20.866534 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc86852-627c-45f6-998c-19dece304a2e admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:20.867013 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc86852-627c-45f6-998c-19dece304a2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:20.867296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc86852-627c-45f6-998c-19dece304a2e admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:20.867977 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc86852-627c-45f6-998c-19dece304a2e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:20.870761 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc86852-627c-45f6-998c-19dece304a2e admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:20.920905 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 895/1794] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:30:20 2022] POST /identity/v3/users => generated 285 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:20.927201 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c19cc6b-dd40-40c9-a7da-d218d5a06b2b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:20.947229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c19cc6b-dd40-40c9-a7da-d218d5a06b2b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:20.955306 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a28f26-65bb-4c71-b53d-8786f69f6bde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:20.956656 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a28f26-65bb-4c71-b53d-8786f69f6bde admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:20.956656 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a28f26-65bb-4c71-b53d-8786f69f6bde admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:20.956970 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a28f26-65bb-4c71-b53d-8786f69f6bde admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:20.957735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a28f26-65bb-4c71-b53d-8786f69f6bde admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:20.960506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a28f26-65bb-4c71-b53d-8786f69f6bde admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:20.976082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 900/1795] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:20 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:30:20.979934 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc86852-627c-45f6-998c-19dece304a2e admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:20.999992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc86852-627c-45f6-998c-19dece304a2e admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:21.010436 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1ca8a9-22f0-4184-9cdf-97104f5a3c27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:21.011169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1ca8a9-22f0-4184-9cdf-97104f5a3c27 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:21.011581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1ca8a9-22f0-4184-9cdf-97104f5a3c27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:21.011933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1ca8a9-22f0-4184-9cdf-97104f5a3c27 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:21.012796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1ca8a9-22f0-4184-9cdf-97104f5a3c27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:21.014899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1ca8a9-22f0-4184-9cdf-97104f5a3c27 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:21.033297 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a28f26-65bb-4c71-b53d-8786f69f6bde admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:21.042721 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 896/1796] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:20 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:21.048038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a28f26-65bb-4c71-b53d-8786f69f6bde admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:21.055602 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5150b472-2bce-4af9-b897-ba3ea24295cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:21.056476 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5150b472-2bce-4af9-b897-ba3ea24295cf admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:21.056829 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5150b472-2bce-4af9-b897-ba3ea24295cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:21.057174 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5150b472-2bce-4af9-b897-ba3ea24295cf admin admin] PATH_INFO: `/v3/projects/5006bacf46684aa9a7b1b5f3d29ee78a/users/eacb5c92a3574921b503d6ead653a66d/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:21.074599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:21.078347 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5150b472-2bce-4af9-b897-ba3ea24295cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=5006bacf46684aa9a7b1b5f3d29ee78a, user_id=eacb5c92a3574921b503d6ead653a66d, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:21.081899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5150b472-2bce-4af9-b897-ba3ea24295cf admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:21.117820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 901/1797] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:21 2022] PUT /identity/v3/projects/5006bacf46684aa9a7b1b5f3d29ee78a/users/eacb5c92a3574921b503d6ead653a66d/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:21.130765 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada1649c-08bb-49ff-8d41-94238e3e321a None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:21.132502 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada1649c-08bb-49ff-8d41-94238e3e321a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:21.132936 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada1649c-08bb-49ff-8d41-94238e3e321a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:21.247536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-ada1649c-08bb-49ff-8d41-94238e3e321a None None] MFA Rules not processed for user `eacb5c92a3574921b503d6ead653a66d`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:21.249229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ada1649c-08bb-49ff-8d41-94238e3e321a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:21.364742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:21.412825 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 897/1798] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:21 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:21.429869 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5150b472-2bce-4af9-b897-ba3ea24295cf admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:21.431916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5150b472-2bce-4af9-b897-ba3ea24295cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:21.631541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5150b472-2bce-4af9-b897-ba3ea24295cf admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:21.635782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea5a716-4391-461d-9f30-b7bd492659a3 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35a5a4ef40204e158649ca25583bbde7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:21.636632 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea5a716-4391-461d-9f30-b7bd492659a3 service glance] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:21.637079 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea5a716-4391-461d-9f30-b7bd492659a3 service glance] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:21.637505 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea5a716-4391-461d-9f30-b7bd492659a3 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:21.648033 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea5a716-4391-461d-9f30-b7bd492659a3 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:21.650013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea5a716-4391-461d-9f30-b7bd492659a3 service glance] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:21.661247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 902/1799] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:21 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:30:29.921625 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada1649c-08bb-49ff-8d41-94238e3e321a None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:29.924145 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ada1649c-08bb-49ff-8d41-94238e3e321a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:30.046162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada1649c-08bb-49ff-8d41-94238e3e321a None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:30.052287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad41690-098e-4ba2-a982-083bf3b4de44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:30.053139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad41690-098e-4ba2-a982-083bf3b4de44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:30.053424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad41690-098e-4ba2-a982-083bf3b4de44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:30.053702 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad41690-098e-4ba2-a982-083bf3b4de44 admin admin] PATH_INFO: `/v3/users/8369731f29994ef18714381ffea041ea` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:30.055075 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad41690-098e-4ba2-a982-083bf3b4de44 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8369731f29994ef18714381ffea041ea)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:30.057945 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad41690-098e-4ba2-a982-083bf3b4de44 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:30.130454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-9ad41690-098e-4ba2-a982-083bf3b4de44 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8369731f29994ef18714381ffea041ea'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:30.132850 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-9ad41690-098e-4ba2-a982-083bf3b4de44 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8369731f29994ef18714381ffea041ea'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:30.133902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-9ad41690-098e-4ba2-a982-083bf3b4de44 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8369731f29994ef18714381ffea041ea'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:30.154355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-9ad41690-098e-4ba2-a982-083bf3b4de44 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8369731f29994ef18714381ffea041ea'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:30.197841 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-9ad41690-098e-4ba2-a982-083bf3b4de44 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8369731f29994ef18714381ffea041ea'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:30.239082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 898/1800] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:30:29 2022] DELETE /identity/v3/users/8369731f29994ef18714381ffea041ea => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:30.246803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e02c229-b5f1-42a6-946c-48920a99af1d None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:30.247435 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e02c229-b5f1-42a6-946c-48920a99af1d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:30.249377 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e02c229-b5f1-42a6-946c-48920a99af1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:30.333151 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-0e02c229-b5f1-42a6-946c-48920a99af1d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:30.339681 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-0e02c229-b5f1-42a6-946c-48920a99af1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:30.527620 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 903/1801] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:30.557424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad41690-098e-4ba2-a982-083bf3b4de44 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:30.565056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9ad41690-098e-4ba2-a982-083bf3b4de44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:30.817228 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad41690-098e-4ba2-a982-083bf3b4de44 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:30.841637 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f448fce1-7f0d-45de-987a-644bf985d77f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:30.842684 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f448fce1-7f0d-45de-987a-644bf985d77f service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:30.843135 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f448fce1-7f0d-45de-987a-644bf985d77f service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:30.843564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f448fce1-7f0d-45de-987a-644bf985d77f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:30.890111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f448fce1-7f0d-45de-987a-644bf985d77f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:30.893521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f448fce1-7f0d-45de-987a-644bf985d77f service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:30.940659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 899/1802] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:30.972826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d676848d-fc88-4c71-b6e9-cee1bcca3cd4 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:30.973238 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d676848d-fc88-4c71-b6e9-cee1bcca3cd4 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:30.973754 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d676848d-fc88-4c71-b6e9-cee1bcca3cd4 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:30.978522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 904/1803] 10.222.0.64 () {54 vars in 921 bytes} [Tue Oct 4 14:30:30 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:30:31.003521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540b6139-47d8-4887-8094-e0d20514aac0 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:31.004082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540b6139-47d8-4887-8094-e0d20514aac0 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:31.004082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540b6139-47d8-4887-8094-e0d20514aac0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:31.029441 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:31.057032 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-540b6139-47d8-4887-8094-e0d20514aac0 None None] MFA Rules not processed for user `03081d7d581745c88c12e9d089fcb2d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:31.061562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-540b6139-47d8-4887-8094-e0d20514aac0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:31.356425 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 900/1804] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Oct 4 14:30:31 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 354 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:30:31.944896 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d676848d-fc88-4c71-b6e9-cee1bcca3cd4 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:31.948366 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d676848d-fc88-4c71-b6e9-cee1bcca3cd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:32.007013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d676848d-fc88-4c71-b6e9-cee1bcca3cd4 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:32.030532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add76c82-2b50-4a47-8c3c-b7d3ebd5ca6d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:32.031957 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add76c82-2b50-4a47-8c3c-b7d3ebd5ca6d service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:32.035825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add76c82-2b50-4a47-8c3c-b7d3ebd5ca6d service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:32.038379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add76c82-2b50-4a47-8c3c-b7d3ebd5ca6d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:32.070307 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add76c82-2b50-4a47-8c3c-b7d3ebd5ca6d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:32.071915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add76c82-2b50-4a47-8c3c-b7d3ebd5ca6d service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:32.120185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 905/1805] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:31 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:32.913609 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540b6139-47d8-4887-8094-e0d20514aac0 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:32.915796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-540b6139-47d8-4887-8094-e0d20514aac0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:32.957238 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540b6139-47d8-4887-8094-e0d20514aac0 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:32.976905 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db782b7-2677-488b-8216-534e272d395f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:32.978218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db782b7-2677-488b-8216-534e272d395f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:32.980882 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db782b7-2677-488b-8216-534e272d395f admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:32.981540 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db782b7-2677-488b-8216-534e272d395f admin admin] PATH_INFO: `/v3/projects/a5acd1c20dbb414588bf9fa0d89cae3a` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:32.983888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db782b7-2677-488b-8216-534e272d395f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a5acd1c20dbb414588bf9fa0d89cae3a)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:32.993899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db782b7-2677-488b-8216-534e272d395f admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:33.122430 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0db782b7-2677-488b-8216-534e272d395f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a5acd1c20dbb414588bf9fa0d89cae3a', 'request_id': 'req-0db782b7-2677-488b-8216-534e272d395f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:33.141703 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0db782b7-2677-488b-8216-534e272d395f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a5acd1c20dbb414588bf9fa0d89cae3a', 'request_id': 'req-0db782b7-2677-488b-8216-534e272d395f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:33.164555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0db782b7-2677-488b-8216-534e272d395f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a5acd1c20dbb414588bf9fa0d89cae3a', 'request_id': 'req-0db782b7-2677-488b-8216-534e272d395f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:33.214690 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0db782b7-2677-488b-8216-534e272d395f admin admin] The token cache is being invalidate because project a5acd1c20dbb414588bf9fa0d89cae3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:30:33.215431 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0db782b7-2677-488b-8216-534e272d395f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:33.220020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0db782b7-2677-488b-8216-534e272d395f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:33.223887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 901/1806] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:30:32 2022] DELETE /identity/v3/projects/a5acd1c20dbb414588bf9fa0d89cae3a => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:33.231304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add76c82-2b50-4a47-8c3c-b7d3ebd5ca6d service nova] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:33.266167 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add76c82-2b50-4a47-8c3c-b7d3ebd5ca6d service nova] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:33.517366 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9319c842-2746-4325-8376-f8a5d5607618 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:33.518438 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9319c842-2746-4325-8376-f8a5d5607618 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:33.519541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9319c842-2746-4325-8376-f8a5d5607618 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:33.520147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9319c842-2746-4325-8376-f8a5d5607618 admin admin] PATH_INFO: `/v3/users/4f81900a54f84c1087dd66257c7d8de8` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:33.523953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9319c842-2746-4325-8376-f8a5d5607618 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4f81900a54f84c1087dd66257c7d8de8)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:33.526778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9319c842-2746-4325-8376-f8a5d5607618 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:33.532093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:33.673955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9319c842-2746-4325-8376-f8a5d5607618 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4f81900a54f84c1087dd66257c7d8de8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:33.689942 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9319c842-2746-4325-8376-f8a5d5607618 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4f81900a54f84c1087dd66257c7d8de8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:33.712755 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9319c842-2746-4325-8376-f8a5d5607618 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4f81900a54f84c1087dd66257c7d8de8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:33.760820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9319c842-2746-4325-8376-f8a5d5607618 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4f81900a54f84c1087dd66257c7d8de8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:33.762012 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9319c842-2746-4325-8376-f8a5d5607618 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4f81900a54f84c1087dd66257c7d8de8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:33.773636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 906/1807] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:30:33 2022] DELETE /identity/v3/users/4f81900a54f84c1087dd66257c7d8de8 => generated 0 bytes in 546 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:34.390697 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db782b7-2677-488b-8216-534e272d395f admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:34.393380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0db782b7-2677-488b-8216-534e272d395f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:34.411439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db782b7-2677-488b-8216-534e272d395f admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:34.507982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b865eb-59d8-4196-b797-d0661fc3798e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:34.508649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b865eb-59d8-4196-b797-d0661fc3798e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:34.508931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b865eb-59d8-4196-b797-d0661fc3798e admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:34.509232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b865eb-59d8-4196-b797-d0661fc3798e admin admin] PATH_INFO: `/v3/projects/66e6a072abf54472b42c30881306fa58` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:34.510497 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b865eb-59d8-4196-b797-d0661fc3798e admin admin] RBAC: Authorizing `identity:delete_project(project_id=66e6a072abf54472b42c30881306fa58)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:34.512571 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b865eb-59d8-4196-b797-d0661fc3798e admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:34.554590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:34.582392 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d0b865eb-59d8-4196-b797-d0661fc3798e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '66e6a072abf54472b42c30881306fa58', 'request_id': 'req-d0b865eb-59d8-4196-b797-d0661fc3798e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:34.589157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d0b865eb-59d8-4196-b797-d0661fc3798e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '66e6a072abf54472b42c30881306fa58', 'request_id': 'req-d0b865eb-59d8-4196-b797-d0661fc3798e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:34.598481 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d0b865eb-59d8-4196-b797-d0661fc3798e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '66e6a072abf54472b42c30881306fa58', 'request_id': 'req-d0b865eb-59d8-4196-b797-d0661fc3798e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:34.616073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d0b865eb-59d8-4196-b797-d0661fc3798e admin admin] The token cache is being invalidate because project 66e6a072abf54472b42c30881306fa58 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:30:34.616649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d0b865eb-59d8-4196-b797-d0661fc3798e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:34.617441 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d0b865eb-59d8-4196-b797-d0661fc3798e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:34.619729 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 902/1808] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:30:34 2022] DELETE /identity/v3/projects/66e6a072abf54472b42c30881306fa58 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:34.623984 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42425de9-8e81-4fd8-ad34-5d2496104c50 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:34.624491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42425de9-8e81-4fd8-ad34-5d2496104c50 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:34.624598 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42425de9-8e81-4fd8-ad34-5d2496104c50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:34.652413 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-42425de9-8e81-4fd8-ad34-5d2496104c50 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:34.654568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-42425de9-8e81-4fd8-ad34-5d2496104c50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:34.763723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 907/1809] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:34.772897 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4358a42-36ed-401e-bed9-dc6e7b45d55e None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:34.773223 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4358a42-36ed-401e-bed9-dc6e7b45d55e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:34.773460 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4358a42-36ed-401e-bed9-dc6e7b45d55e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:34.799045 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-b4358a42-36ed-401e-bed9-dc6e7b45d55e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:34.800490 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-b4358a42-36ed-401e-bed9-dc6e7b45d55e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:34.825071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42425de9-8e81-4fd8-ad34-5d2496104c50 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:34.832861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 903/1810] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:34.837908 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42425de9-8e81-4fd8-ad34-5d2496104c50 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:34.846323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f0974f-1bf7-4f49-89eb-790433305102 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:34.846958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f0974f-1bf7-4f49-89eb-790433305102 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:34.847236 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f0974f-1bf7-4f49-89eb-790433305102 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:34.847648 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f0974f-1bf7-4f49-89eb-790433305102 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:34.848323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f0974f-1bf7-4f49-89eb-790433305102 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:30:34.848643 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f0974f-1bf7-4f49-89eb-790433305102 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:34.850994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f0974f-1bf7-4f49-89eb-790433305102 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:34.877145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 908/1811] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:30:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:30:34.880993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4358a42-36ed-401e-bed9-dc6e7b45d55e None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:34.896801 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4358a42-36ed-401e-bed9-dc6e7b45d55e None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:34.902574 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c71fe5-4b1b-413f-b65e-726185008827 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:34.903213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c71fe5-4b1b-413f-b65e-726185008827 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:34.903492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c71fe5-4b1b-413f-b65e-726185008827 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:34.903764 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c71fe5-4b1b-413f-b65e-726185008827 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:34.904479 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c71fe5-4b1b-413f-b65e-726185008827 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:34.906421 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c71fe5-4b1b-413f-b65e-726185008827 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:34.940351 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 904/1812] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:30:34 2022] POST /identity/v3/projects => generated 385 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:30:34.943765 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f0974f-1bf7-4f49-89eb-790433305102 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:34.954970 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f0974f-1bf7-4f49-89eb-790433305102 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:35.033674 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bf041e-e86f-4223-ad37-65742c54f0e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:35.035687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bf041e-e86f-4223-ad37-65742c54f0e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:35.035687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bf041e-e86f-4223-ad37-65742c54f0e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:35.035687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bf041e-e86f-4223-ad37-65742c54f0e9 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:35.035687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bf041e-e86f-4223-ad37-65742c54f0e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:35.038536 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bf041e-e86f-4223-ad37-65742c54f0e9 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:35.061384 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:35.082293 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 909/1813] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:30:34 2022] POST /identity/v3/users => generated 293 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:30:35.088848 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c71fe5-4b1b-413f-b65e-726185008827 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:35.103617 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c71fe5-4b1b-413f-b65e-726185008827 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:35.112434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c828d3a9-961c-43c3-9552-fae78e37e0c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:35.112434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c828d3a9-961c-43c3-9552-fae78e37e0c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:35.112434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c828d3a9-961c-43c3-9552-fae78e37e0c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:35.112434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c828d3a9-961c-43c3-9552-fae78e37e0c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:35.113297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c828d3a9-961c-43c3-9552-fae78e37e0c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:35.115005 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c828d3a9-961c-43c3-9552-fae78e37e0c0 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:35.126765 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 905/1814] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:35 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:35.131421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bf041e-e86f-4223-ad37-65742c54f0e9 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:35.149639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bf041e-e86f-4223-ad37-65742c54f0e9 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:35.157904 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4505ba35-5c62-408a-874f-014eb36c58f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:35.158803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4505ba35-5c62-408a-874f-014eb36c58f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:35.159088 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4505ba35-5c62-408a-874f-014eb36c58f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:35.159361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4505ba35-5c62-408a-874f-014eb36c58f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:35.160334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4505ba35-5c62-408a-874f-014eb36c58f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:35.162509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4505ba35-5c62-408a-874f-014eb36c58f3 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:35.179086 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 910/1815] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:35 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:35.183252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c828d3a9-961c-43c3-9552-fae78e37e0c0 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:35.195838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c828d3a9-961c-43c3-9552-fae78e37e0c0 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:35.212236 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1c7053-e421-4d44-ba4e-10a8ce59c9ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:35.212236 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1c7053-e421-4d44-ba4e-10a8ce59c9ec admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:35.212531 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1c7053-e421-4d44-ba4e-10a8ce59c9ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:35.212770 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1c7053-e421-4d44-ba4e-10a8ce59c9ec admin admin] PATH_INFO: `/v3/projects/3a8972134f1b4e958c4133695eaf6042/users/9eefa341ebe84314aab7707a040aa2af/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:35.242243 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1c7053-e421-4d44-ba4e-10a8ce59c9ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a8972134f1b4e958c4133695eaf6042, user_id=9eefa341ebe84314aab7707a040aa2af, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:35.245247 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1c7053-e421-4d44-ba4e-10a8ce59c9ec admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:35.279976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 906/1816] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:35 2022] PUT /identity/v3/projects/3a8972134f1b4e958c4133695eaf6042/users/9eefa341ebe84314aab7707a040aa2af/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:35.285040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe07d65-6417-4d38-be90-78c533cf6602 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:35.285324 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe07d65-6417-4d38-be90-78c533cf6602 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:35.285605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe07d65-6417-4d38-be90-78c533cf6602 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:35.356404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-1fe07d65-6417-4d38-be90-78c533cf6602 None None] MFA Rules not processed for user `9eefa341ebe84314aab7707a040aa2af`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:35.358539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1fe07d65-6417-4d38-be90-78c533cf6602 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:35.495957 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 911/1817] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:35 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:35.500661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1c7053-e421-4d44-ba4e-10a8ce59c9ec admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:35.590518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1c7053-e421-4d44-ba4e-10a8ce59c9ec admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:35.599835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9a93bf-901b-434a-94ce-adbef3d2fc05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:35.600044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9a93bf-901b-434a-94ce-adbef3d2fc05 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:35.600208 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9a93bf-901b-434a-94ce-adbef3d2fc05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:35.600527 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9a93bf-901b-434a-94ce-adbef3d2fc05 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:35.601186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9a93bf-901b-434a-94ce-adbef3d2fc05 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:35.603113 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9a93bf-901b-434a-94ce-adbef3d2fc05 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:35.635710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 907/1818] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:30:35 2022] POST /identity/v3/projects => generated 387 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:35.641005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe07d65-6417-4d38-be90-78c533cf6602 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:35.658472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe07d65-6417-4d38-be90-78c533cf6602 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:35.724008 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba0db1b-73ae-4788-9ffa-b1eb02ec8b50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:35.725551 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba0db1b-73ae-4788-9ffa-b1eb02ec8b50 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:35.726036 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba0db1b-73ae-4788-9ffa-b1eb02ec8b50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:35.726326 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba0db1b-73ae-4788-9ffa-b1eb02ec8b50 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:35.727215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba0db1b-73ae-4788-9ffa-b1eb02ec8b50 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:35.729683 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba0db1b-73ae-4788-9ffa-b1eb02ec8b50 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:35.754722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 912/1819] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:30:35 2022] POST /identity/v3/users => generated 300 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:35.760428 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9a93bf-901b-434a-94ce-adbef3d2fc05 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:35.774699 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9a93bf-901b-434a-94ce-adbef3d2fc05 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:35.780185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42b1c15-689a-4890-ae98-400c3c49417d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:35.780846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42b1c15-689a-4890-ae98-400c3c49417d admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:35.781124 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42b1c15-689a-4890-ae98-400c3c49417d admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:35.781395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42b1c15-689a-4890-ae98-400c3c49417d admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:35.782043 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42b1c15-689a-4890-ae98-400c3c49417d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:35.783984 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42b1c15-689a-4890-ae98-400c3c49417d admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:35.800735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba0db1b-73ae-4788-9ffa-b1eb02ec8b50 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:35.801753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 908/1820] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:35 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:35.812417 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba0db1b-73ae-4788-9ffa-b1eb02ec8b50 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:35.818870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c747984-567e-48fc-8538-f47577022a98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:35.819557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c747984-567e-48fc-8538-f47577022a98 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:35.819835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c747984-567e-48fc-8538-f47577022a98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:35.820108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c747984-567e-48fc-8538-f47577022a98 admin admin] PATH_INFO: `/v3/projects/b2d952ad64314600a876e74eac87a424/users/cc95e12ed18f41a9b80a6a629add66c8/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:35.842148 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c747984-567e-48fc-8538-f47577022a98 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b2d952ad64314600a876e74eac87a424, user_id=cc95e12ed18f41a9b80a6a629add66c8, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:35.845224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c747984-567e-48fc-8538-f47577022a98 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:35.861733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 913/1821] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:35 2022] PUT /identity/v3/projects/b2d952ad64314600a876e74eac87a424/users/cc95e12ed18f41a9b80a6a629add66c8/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:30:35.866553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42b1c15-689a-4890-ae98-400c3c49417d admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:35.879196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42b1c15-689a-4890-ae98-400c3c49417d admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:35.915008 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=90,keystone:DELETE=6,keystone:INSERT=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:35.996902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c915d9-ed3c-413f-9206-7ef7280b6cb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:35.997700 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c915d9-ed3c-413f-9206-7ef7280b6cb9 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:35.998042 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c915d9-ed3c-413f-9206-7ef7280b6cb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:35.998390 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c915d9-ed3c-413f-9206-7ef7280b6cb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:35.999150 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c915d9-ed3c-413f-9206-7ef7280b6cb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:36.001491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c915d9-ed3c-413f-9206-7ef7280b6cb9 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:36.020219 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 909/1822] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:35 2022] GET /identity/v3/roles => generated 1420 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:36.025213 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c747984-567e-48fc-8538-f47577022a98 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:36.040625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c747984-567e-48fc-8538-f47577022a98 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:36.051197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b17e7f-48ea-4649-81d0-1b0f71ae5ae5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:36.051890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b17e7f-48ea-4649-81d0-1b0f71ae5ae5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:36.052249 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b17e7f-48ea-4649-81d0-1b0f71ae5ae5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:36.052751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b17e7f-48ea-4649-81d0-1b0f71ae5ae5 admin admin] PATH_INFO: `/v3/projects/b2d952ad64314600a876e74eac87a424/users/cc95e12ed18f41a9b80a6a629add66c8/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:36.056270 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b17e7f-48ea-4649-81d0-1b0f71ae5ae5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b2d952ad64314600a876e74eac87a424, user_id=cc95e12ed18f41a9b80a6a629add66c8, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:36.059095 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b17e7f-48ea-4649-81d0-1b0f71ae5ae5 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:36.074362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 914/1823] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:36 2022] PUT /identity/v3/projects/b2d952ad64314600a876e74eac87a424/users/cc95e12ed18f41a9b80a6a629add66c8/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:36.087436 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683e796a-23ce-465d-b061-6fea95492d85 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:36.087728 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683e796a-23ce-465d-b061-6fea95492d85 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:36.088000 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683e796a-23ce-465d-b061-6fea95492d85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:36.133990 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-683e796a-23ce-465d-b061-6fea95492d85 None None] MFA Rules not processed for user `cc95e12ed18f41a9b80a6a629add66c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:36.135523 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-683e796a-23ce-465d-b061-6fea95492d85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:36.214742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 910/1824] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:36 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:36.227251 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b17e7f-48ea-4649-81d0-1b0f71ae5ae5 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:36.229233 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-43b17e7f-48ea-4649-81d0-1b0f71ae5ae5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:36.267779 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:36.299715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b17e7f-48ea-4649-81d0-1b0f71ae5ae5 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:36.305976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a851cbb-c950-4fb1-a3a7-305fc986575c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:36.306721 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a851cbb-c950-4fb1-a3a7-305fc986575c service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:36.307089 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a851cbb-c950-4fb1-a3a7-305fc986575c service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:36.307450 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a851cbb-c950-4fb1-a3a7-305fc986575c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:36.318467 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a851cbb-c950-4fb1-a3a7-305fc986575c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:36.320148 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a851cbb-c950-4fb1-a3a7-305fc986575c service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:36.331039 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 915/1825] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:36 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:36.365718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683e796a-23ce-465d-b061-6fea95492d85 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:36.467387 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683e796a-23ce-465d-b061-6fea95492d85 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:36.472616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b0d8a3-0176-4c8e-8064-1e14d6f099ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:36.473294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b0d8a3-0176-4c8e-8064-1e14d6f099ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:36.473567 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b0d8a3-0176-4c8e-8064-1e14d6f099ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:36.473846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b0d8a3-0176-4c8e-8064-1e14d6f099ab admin admin] PATH_INFO: `/v3/users/9eefa341ebe84314aab7707a040aa2af` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:36.475550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b0d8a3-0176-4c8e-8064-1e14d6f099ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=9eefa341ebe84314aab7707a040aa2af)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:36.477405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b0d8a3-0176-4c8e-8064-1e14d6f099ab admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:36.554879 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:36.557330 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d3b0d8a3-0176-4c8e-8064-1e14d6f099ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9eefa341ebe84314aab7707a040aa2af'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:36.558228 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d3b0d8a3-0176-4c8e-8064-1e14d6f099ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9eefa341ebe84314aab7707a040aa2af'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:36.559175 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d3b0d8a3-0176-4c8e-8064-1e14d6f099ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9eefa341ebe84314aab7707a040aa2af'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:36.574208 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d3b0d8a3-0176-4c8e-8064-1e14d6f099ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9eefa341ebe84314aab7707a040aa2af'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:36.592254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d3b0d8a3-0176-4c8e-8064-1e14d6f099ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9eefa341ebe84314aab7707a040aa2af'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:36.620861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 911/1826] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:30:36 2022] DELETE /identity/v3/users/9eefa341ebe84314aab7707a040aa2af => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:36.624230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43641576-5988-46ff-8ddd-b82e562d2660 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:36.624538 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43641576-5988-46ff-8ddd-b82e562d2660 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:36.624805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43641576-5988-46ff-8ddd-b82e562d2660 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:36.713046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-43641576-5988-46ff-8ddd-b82e562d2660 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:36.718469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-43641576-5988-46ff-8ddd-b82e562d2660 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:36.820504 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 916/1827] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:36.829702 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b0d8a3-0176-4c8e-8064-1e14d6f099ab admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:36.831323 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d3b0d8a3-0176-4c8e-8064-1e14d6f099ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:36.909079 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b0d8a3-0176-4c8e-8064-1e14d6f099ab admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:36.916909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4031ad10-1c56-40a3-af53-6c2572d38a2c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:36.917521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4031ad10-1c56-40a3-af53-6c2572d38a2c service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:36.917838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4031ad10-1c56-40a3-af53-6c2572d38a2c service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:36.918207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4031ad10-1c56-40a3-af53-6c2572d38a2c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:36.933306 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4031ad10-1c56-40a3-af53-6c2572d38a2c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:36.935357 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4031ad10-1c56-40a3-af53-6c2572d38a2c service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:36.955978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 912/1828] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:37.437195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43641576-5988-46ff-8ddd-b82e562d2660 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:37.438724 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-43641576-5988-46ff-8ddd-b82e562d2660 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:37.453656 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43641576-5988-46ff-8ddd-b82e562d2660 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:37.461640 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:37.462501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:37.463294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:37.463680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f admin admin] PATH_INFO: `/v3/projects/3a8972134f1b4e958c4133695eaf6042` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:37.465306 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a8972134f1b4e958c4133695eaf6042)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:37.467400 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:37.530426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a8972134f1b4e958c4133695eaf6042', 'request_id': 'req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:37.539119 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a8972134f1b4e958c4133695eaf6042', 'request_id': 'req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:37.550005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a8972134f1b4e958c4133695eaf6042', 'request_id': 'req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:37.568282 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f admin admin] The token cache is being invalidate because project 3a8972134f1b4e958c4133695eaf6042 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:30:37.571391 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:37.572364 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:37.574569 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 917/1829] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:30:37 2022] DELETE /identity/v3/projects/3a8972134f1b4e958c4133695eaf6042 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:37.580174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4031ad10-1c56-40a3-af53-6c2572d38a2c service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:37.612911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4031ad10-1c56-40a3-af53-6c2572d38a2c service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:37.726274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735bbee4-5cbe-41b3-ad6b-f2fa78352ce8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:37.729072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735bbee4-5cbe-41b3-ad6b-f2fa78352ce8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:37.729446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735bbee4-5cbe-41b3-ad6b-f2fa78352ce8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:37.729635 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735bbee4-5cbe-41b3-ad6b-f2fa78352ce8 admin admin] PATH_INFO: `/v3/users/cc95e12ed18f41a9b80a6a629add66c8` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:37.732146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735bbee4-5cbe-41b3-ad6b-f2fa78352ce8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cc95e12ed18f41a9b80a6a629add66c8)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:37.733361 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735bbee4-5cbe-41b3-ad6b-f2fa78352ce8 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:37.837271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-735bbee4-5cbe-41b3-ad6b-f2fa78352ce8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cc95e12ed18f41a9b80a6a629add66c8'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:37.838163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-735bbee4-5cbe-41b3-ad6b-f2fa78352ce8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cc95e12ed18f41a9b80a6a629add66c8'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:37.838918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-735bbee4-5cbe-41b3-ad6b-f2fa78352ce8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cc95e12ed18f41a9b80a6a629add66c8'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:37.854233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-735bbee4-5cbe-41b3-ad6b-f2fa78352ce8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cc95e12ed18f41a9b80a6a629add66c8'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:37.887245 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=86,keystone:INSERT=2,keystone:DELETE=12 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:37.894570 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-735bbee4-5cbe-41b3-ad6b-f2fa78352ce8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cc95e12ed18f41a9b80a6a629add66c8'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:37.927444 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 913/1830] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:30:37 2022] DELETE /identity/v3/users/cc95e12ed18f41a9b80a6a629add66c8 => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:39.275953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:39.277469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:39.300976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a00f06-a8a8-4d0d-a3a9-f513d1ee7e4f admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:39.450437 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:39.472176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fc2fb0-799a-4642-9c2c-6458576a5eaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:39.472902 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fc2fb0-799a-4642-9c2c-6458576a5eaf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:39.473247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fc2fb0-799a-4642-9c2c-6458576a5eaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:39.473449 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fc2fb0-799a-4642-9c2c-6458576a5eaf admin admin] PATH_INFO: `/v3/projects/b2d952ad64314600a876e74eac87a424` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:39.475369 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fc2fb0-799a-4642-9c2c-6458576a5eaf admin admin] RBAC: Authorizing `identity:delete_project(project_id=b2d952ad64314600a876e74eac87a424)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:39.477654 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fc2fb0-799a-4642-9c2c-6458576a5eaf admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:39.583385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-06fc2fb0-799a-4642-9c2c-6458576a5eaf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b2d952ad64314600a876e74eac87a424', 'request_id': 'req-06fc2fb0-799a-4642-9c2c-6458576a5eaf'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:39.603860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-06fc2fb0-799a-4642-9c2c-6458576a5eaf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b2d952ad64314600a876e74eac87a424', 'request_id': 'req-06fc2fb0-799a-4642-9c2c-6458576a5eaf'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:39.619356 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-06fc2fb0-799a-4642-9c2c-6458576a5eaf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b2d952ad64314600a876e74eac87a424', 'request_id': 'req-06fc2fb0-799a-4642-9c2c-6458576a5eaf'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:39.643113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-06fc2fb0-799a-4642-9c2c-6458576a5eaf admin admin] The token cache is being invalidate because project b2d952ad64314600a876e74eac87a424 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:30:39.644143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-06fc2fb0-799a-4642-9c2c-6458576a5eaf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:39.647346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-06fc2fb0-799a-4642-9c2c-6458576a5eaf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:39.650457 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 918/1831] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:30:39 2022] DELETE /identity/v3/projects/b2d952ad64314600a876e74eac87a424 => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:39.655733 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadaa332-15de-418a-8798-0fdc3bc9a11c None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:39.656620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadaa332-15de-418a-8798-0fdc3bc9a11c None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:39.659462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadaa332-15de-418a-8798-0fdc3bc9a11c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:39.689553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-cadaa332-15de-418a-8798-0fdc3bc9a11c None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:39.690991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-cadaa332-15de-418a-8798-0fdc3bc9a11c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:39.804470 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 914/1832] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:39.813609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a573a145-aaa9-4231-8992-cebb78877220 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:39.814684 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a573a145-aaa9-4231-8992-cebb78877220 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:39.814684 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a573a145-aaa9-4231-8992-cebb78877220 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:39.842014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-a573a145-aaa9-4231-8992-cebb78877220 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:39.843347 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a573a145-aaa9-4231-8992-cebb78877220 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:39.853608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 919/1833] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:39.861287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadaa332-15de-418a-8798-0fdc3bc9a11c None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:39.878056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadaa332-15de-418a-8798-0fdc3bc9a11c None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:39.884612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4108082-8437-4723-9570-83d803492104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:39.885303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4108082-8437-4723-9570-83d803492104 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:39.885586 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4108082-8437-4723-9570-83d803492104 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:39.885859 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4108082-8437-4723-9570-83d803492104 admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:39.886684 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4108082-8437-4723-9570-83d803492104 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:30:39.886994 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4108082-8437-4723-9570-83d803492104 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:39.888937 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4108082-8437-4723-9570-83d803492104 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:39.913747 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 915/1834] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:30:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:30:39.923230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a573a145-aaa9-4231-8992-cebb78877220 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:39.959885 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a573a145-aaa9-4231-8992-cebb78877220 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:39.979593 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bfa100-ee02-414e-a299-63e47dd72269 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:39.980282 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bfa100-ee02-414e-a299-63e47dd72269 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:39.980665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bfa100-ee02-414e-a299-63e47dd72269 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:39.981010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bfa100-ee02-414e-a299-63e47dd72269 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:39.981722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bfa100-ee02-414e-a299-63e47dd72269 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:39.983762 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bfa100-ee02-414e-a299-63e47dd72269 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:40.076413 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 920/1835] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:30:39 2022] POST /identity/v3/projects => generated 365 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:40.078710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4108082-8437-4723-9570-83d803492104 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:40.101134 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4108082-8437-4723-9570-83d803492104 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:40.250627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a884e7d9-b895-4fee-b71b-a0c86e27a0f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:40.251158 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a884e7d9-b895-4fee-b71b-a0c86e27a0f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:40.251445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a884e7d9-b895-4fee-b71b-a0c86e27a0f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:40.251807 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a884e7d9-b895-4fee-b71b-a0c86e27a0f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:40.252379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a884e7d9-b895-4fee-b71b-a0c86e27a0f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:40.254327 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a884e7d9-b895-4fee-b71b-a0c86e27a0f5 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:40.311810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 916/1836] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:30:40 2022] POST /identity/v3/users => generated 283 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:30:40.316996 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bfa100-ee02-414e-a299-63e47dd72269 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:40.336730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bfa100-ee02-414e-a299-63e47dd72269 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:40.344184 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9bace7-5bc7-4647-98f8-82a3ad527807 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:40.345728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9bace7-5bc7-4647-98f8-82a3ad527807 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:40.346320 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9bace7-5bc7-4647-98f8-82a3ad527807 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:40.347249 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9bace7-5bc7-4647-98f8-82a3ad527807 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:40.348612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9bace7-5bc7-4647-98f8-82a3ad527807 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:40.351972 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9bace7-5bc7-4647-98f8-82a3ad527807 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:40.401701 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 921/1837] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:40 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:40.406238 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a884e7d9-b895-4fee-b71b-a0c86e27a0f5 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:40.424878 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a884e7d9-b895-4fee-b71b-a0c86e27a0f5 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:40.441521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932ec7d3-37d3-45a7-86b1-7d0df11a9d92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:40.445058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932ec7d3-37d3-45a7-86b1-7d0df11a9d92 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:40.445058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932ec7d3-37d3-45a7-86b1-7d0df11a9d92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:40.445058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932ec7d3-37d3-45a7-86b1-7d0df11a9d92 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:40.445688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932ec7d3-37d3-45a7-86b1-7d0df11a9d92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:40.452340 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932ec7d3-37d3-45a7-86b1-7d0df11a9d92 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:40.478022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9bace7-5bc7-4647-98f8-82a3ad527807 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:40.481180 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 917/1838] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:40 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:40.514938 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9bace7-5bc7-4647-98f8-82a3ad527807 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:40.536123 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1c0500-293e-440f-96eb-0487e9d0a4f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:40.538858 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1c0500-293e-440f-96eb-0487e9d0a4f0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:40.539147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1c0500-293e-440f-96eb-0487e9d0a4f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:40.539289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1c0500-293e-440f-96eb-0487e9d0a4f0 admin admin] PATH_INFO: `/v3/projects/b513e7d714ca432b8e9e83fe9be608f1/users/78520885bf274596af992937a9c1a397/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:40.560459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1c0500-293e-440f-96eb-0487e9d0a4f0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b513e7d714ca432b8e9e83fe9be608f1, user_id=78520885bf274596af992937a9c1a397, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:40.562570 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1c0500-293e-440f-96eb-0487e9d0a4f0 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:40.587446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 922/1839] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:40 2022] PUT /identity/v3/projects/b513e7d714ca432b8e9e83fe9be608f1/users/78520885bf274596af992937a9c1a397/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:40.597716 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f764b70-11be-4d90-a11a-69d4a3d742bf None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:40.598006 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f764b70-11be-4d90-a11a-69d4a3d742bf None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:40.598292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f764b70-11be-4d90-a11a-69d4a3d742bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:40.637967 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-7f764b70-11be-4d90-a11a-69d4a3d742bf None None] MFA Rules not processed for user `78520885bf274596af992937a9c1a397`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:40.639228 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7f764b70-11be-4d90-a11a-69d4a3d742bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:40.726117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:DELETE=1,keystone:SELECT=96,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:40.776662 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 918/1840] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:40 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:40.780654 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1c0500-293e-440f-96eb-0487e9d0a4f0 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:40.882453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1c0500-293e-440f-96eb-0487e9d0a4f0 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:40.902232 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfb8e39-bcc7-4137-9f02-2ab860c84021 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:40.903138 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfb8e39-bcc7-4137-9f02-2ab860c84021 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:40.903604 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfb8e39-bcc7-4137-9f02-2ab860c84021 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:40.903919 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfb8e39-bcc7-4137-9f02-2ab860c84021 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:40.905103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfb8e39-bcc7-4137-9f02-2ab860c84021 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:40.914609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfb8e39-bcc7-4137-9f02-2ab860c84021 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:40.968257 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 923/1841] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:30:40 2022] POST /identity/v3/projects => generated 367 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:40.971674 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f764b70-11be-4d90-a11a-69d4a3d742bf None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:41.003768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f764b70-11be-4d90-a11a-69d4a3d742bf None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:41.158691 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18804a36-31a8-4246-8ab9-42a4eda4a7f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:41.159531 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18804a36-31a8-4246-8ab9-42a4eda4a7f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:41.159904 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18804a36-31a8-4246-8ab9-42a4eda4a7f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:41.160273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18804a36-31a8-4246-8ab9-42a4eda4a7f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:41.161423 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18804a36-31a8-4246-8ab9-42a4eda4a7f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:41.164050 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18804a36-31a8-4246-8ab9-42a4eda4a7f1 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:41.199303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 919/1842] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:30:40 2022] POST /identity/v3/users => generated 290 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:41.200760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfb8e39-bcc7-4137-9f02-2ab860c84021 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:41.215000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfb8e39-bcc7-4137-9f02-2ab860c84021 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:41.224826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97df4a35-2eda-4e54-8be1-fc6588af93ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:41.226420 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97df4a35-2eda-4e54-8be1-fc6588af93ed admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:41.226889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97df4a35-2eda-4e54-8be1-fc6588af93ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:41.227425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97df4a35-2eda-4e54-8be1-fc6588af93ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:41.228747 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97df4a35-2eda-4e54-8be1-fc6588af93ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:41.231593 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97df4a35-2eda-4e54-8be1-fc6588af93ed admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:41.234932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:41.280805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 924/1843] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:41 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:41.282051 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18804a36-31a8-4246-8ab9-42a4eda4a7f1 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:41.297948 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18804a36-31a8-4246-8ab9-42a4eda4a7f1 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:41.306836 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d250a1a0-e5cd-49bb-bd77-e93a760bb9c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:41.309028 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d250a1a0-e5cd-49bb-bd77-e93a760bb9c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:41.309479 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d250a1a0-e5cd-49bb-bd77-e93a760bb9c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:41.309793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d250a1a0-e5cd-49bb-bd77-e93a760bb9c6 admin admin] PATH_INFO: `/v3/projects/7ab22ef2ab16440cb2d08e6d141e14ed/users/21f47f5bd8b440f49e60d49f7a7b726d/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:41.351362 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d250a1a0-e5cd-49bb-bd77-e93a760bb9c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7ab22ef2ab16440cb2d08e6d141e14ed, user_id=21f47f5bd8b440f49e60d49f7a7b726d, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:41.359081 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d250a1a0-e5cd-49bb-bd77-e93a760bb9c6 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:41.409645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 920/1844] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:41 2022] PUT /identity/v3/projects/7ab22ef2ab16440cb2d08e6d141e14ed/users/21f47f5bd8b440f49e60d49f7a7b726d/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:41.417336 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97df4a35-2eda-4e54-8be1-fc6588af93ed admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:41.469440 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97df4a35-2eda-4e54-8be1-fc6588af93ed admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:41.703706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737a9676-5b7d-4383-868b-c20365827eaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:41.704558 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737a9676-5b7d-4383-868b-c20365827eaf admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:41.704668 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737a9676-5b7d-4383-868b-c20365827eaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:41.704946 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737a9676-5b7d-4383-868b-c20365827eaf admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:41.705587 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737a9676-5b7d-4383-868b-c20365827eaf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:41.710065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737a9676-5b7d-4383-868b-c20365827eaf admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:41.744324 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 925/1845] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:41 2022] GET /identity/v3/roles => generated 1420 bytes in 333 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:41.756078 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d250a1a0-e5cd-49bb-bd77-e93a760bb9c6 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:41.821819 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d250a1a0-e5cd-49bb-bd77-e93a760bb9c6 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:41.828790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12c900b-2a65-403d-b527-a8ac518baa35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:41.829903 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12c900b-2a65-403d-b527-a8ac518baa35 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:41.830354 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12c900b-2a65-403d-b527-a8ac518baa35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:41.831283 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12c900b-2a65-403d-b527-a8ac518baa35 admin admin] PATH_INFO: `/v3/projects/7ab22ef2ab16440cb2d08e6d141e14ed/users/21f47f5bd8b440f49e60d49f7a7b726d/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:41.838051 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12c900b-2a65-403d-b527-a8ac518baa35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7ab22ef2ab16440cb2d08e6d141e14ed, user_id=21f47f5bd8b440f49e60d49f7a7b726d, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:41.868436 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12c900b-2a65-403d-b527-a8ac518baa35 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:41.913180 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 921/1846] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:41 2022] PUT /identity/v3/projects/7ab22ef2ab16440cb2d08e6d141e14ed/users/21f47f5bd8b440f49e60d49f7a7b726d/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:41.924835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7682b05-a572-42b7-bbe8-586456c8dccf None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:41.925137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7682b05-a572-42b7-bbe8-586456c8dccf None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:41.925422 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7682b05-a572-42b7-bbe8-586456c8dccf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:41.977981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-a7682b05-a572-42b7-bbe8-586456c8dccf None None] MFA Rules not processed for user `21f47f5bd8b440f49e60d49f7a7b726d`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:41.979537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a7682b05-a572-42b7-bbe8-586456c8dccf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:42.028891 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12c900b-2a65-403d-b527-a8ac518baa35 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:42.031842 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c12c900b-2a65-403d-b527-a8ac518baa35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:42.123720 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 926/1847] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:41 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:42.146332 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7682b05-a572-42b7-bbe8-586456c8dccf None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:42.147629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12c900b-2a65-403d-b527-a8ac518baa35 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:42.148005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a7682b05-a572-42b7-bbe8-586456c8dccf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:42.156364 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcbd2e79-6f2f-4298-83f9-a5f5697d6ab0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:42.156364 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbd2e79-6f2f-4298-83f9-a5f5697d6ab0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:42.156364 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbd2e79-6f2f-4298-83f9-a5f5697d6ab0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:42.156364 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbd2e79-6f2f-4298-83f9-a5f5697d6ab0 admin admin] PATH_INFO: `/v3/users/3f421d1a9ba04b188a753b862a2b5c7b` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:42.156877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcbd2e79-6f2f-4298-83f9-a5f5697d6ab0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3f421d1a9ba04b188a753b862a2b5c7b)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:42.158037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcbd2e79-6f2f-4298-83f9-a5f5697d6ab0 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:42.209018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:42.226103 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-dcbd2e79-6f2f-4298-83f9-a5f5697d6ab0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3f421d1a9ba04b188a753b862a2b5c7b'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:42.227223 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-dcbd2e79-6f2f-4298-83f9-a5f5697d6ab0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3f421d1a9ba04b188a753b862a2b5c7b'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:42.228236 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-dcbd2e79-6f2f-4298-83f9-a5f5697d6ab0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3f421d1a9ba04b188a753b862a2b5c7b'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:42.240664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7682b05-a572-42b7-bbe8-586456c8dccf None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:42.241434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-dcbd2e79-6f2f-4298-83f9-a5f5697d6ab0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3f421d1a9ba04b188a753b862a2b5c7b'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:42.245316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b9b571-ec47-4f6f-989b-d23b8d7d8fe1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:42.245973 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b9b571-ec47-4f6f-989b-d23b8d7d8fe1 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:42.246248 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b9b571-ec47-4f6f-989b-d23b8d7d8fe1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:42.246520 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b9b571-ec47-4f6f-989b-d23b8d7d8fe1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:42.248512 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-37b9b571-ec47-4f6f-989b-d23b8d7d8fe1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:42.257551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-dcbd2e79-6f2f-4298-83f9-a5f5697d6ab0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3f421d1a9ba04b188a753b862a2b5c7b'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:42.260936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b9b571-ec47-4f6f-989b-d23b8d7d8fe1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:42.262383 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b9b571-ec47-4f6f-989b-d23b8d7d8fe1 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:42.272671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 922/1848] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:30:42 2022] DELETE /identity/v3/users/3f421d1a9ba04b188a753b862a2b5c7b => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:42.277539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 927/1849] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:42 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:42.278071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fe45c2-9559-4eb8-b2df-e7378993917c None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:42.278359 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fe45c2-9559-4eb8-b2df-e7378993917c None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:42.278719 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fe45c2-9559-4eb8-b2df-e7378993917c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:42.302448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-54fe45c2-9559-4eb8-b2df-e7378993917c None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:42.303998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-54fe45c2-9559-4eb8-b2df-e7378993917c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:42.372901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b9b571-ec47-4f6f-989b-d23b8d7d8fe1 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:42.375295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-37b9b571-ec47-4f6f-989b-d23b8d7d8fe1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:42.392198 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b9b571-ec47-4f6f-989b-d23b8d7d8fe1 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:42.398772 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 923/1850] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:42.409440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fe45c2-9559-4eb8-b2df-e7378993917c None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:42.410940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-54fe45c2-9559-4eb8-b2df-e7378993917c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:42.418021 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:42.477544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e073698-d793-4fdf-9f9a-6c75af2ff21a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:42.478980 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e073698-d793-4fdf-9f9a-6c75af2ff21a service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:42.479710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e073698-d793-4fdf-9f9a-6c75af2ff21a service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:42.480104 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e073698-d793-4fdf-9f9a-6c75af2ff21a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:42.482943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-0e073698-d793-4fdf-9f9a-6c75af2ff21a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:42.497451 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fe45c2-9559-4eb8-b2df-e7378993917c None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:42.503514 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21556988-3215-4bbf-beea-9c08693b84da service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:42.504143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21556988-3215-4bbf-beea-9c08693b84da service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:42.504573 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21556988-3215-4bbf-beea-9c08693b84da service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:42.504734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21556988-3215-4bbf-beea-9c08693b84da service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:42.514800 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21556988-3215-4bbf-beea-9c08693b84da service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:42.516651 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21556988-3215-4bbf-beea-9c08693b84da service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:42.527079 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 924/1851] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:42.570205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e073698-d793-4fdf-9f9a-6c75af2ff21a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:42.571852 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e073698-d793-4fdf-9f9a-6c75af2ff21a service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:42.585879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 928/1852] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:42 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:42.908777 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21556988-3215-4bbf-beea-9c08693b84da service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:42.910698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-21556988-3215-4bbf-beea-9c08693b84da service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:42.965075 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21556988-3215-4bbf-beea-9c08693b84da service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:42.971751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b50316-cdda-4265-9d83-8c7f885a6c82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:42.972271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b50316-cdda-4265-9d83-8c7f885a6c82 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:42.972587 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b50316-cdda-4265-9d83-8c7f885a6c82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:42.972859 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b50316-cdda-4265-9d83-8c7f885a6c82 admin admin] PATH_INFO: `/v3/projects/2e3d973693cc45d0abaa391442971e88` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:42.974351 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b50316-cdda-4265-9d83-8c7f885a6c82 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e3d973693cc45d0abaa391442971e88)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:42.977834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b50316-cdda-4265-9d83-8c7f885a6c82 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:43.085710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-61b50316-cdda-4265-9d83-8c7f885a6c82 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e3d973693cc45d0abaa391442971e88', 'request_id': 'req-61b50316-cdda-4265-9d83-8c7f885a6c82'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:43.085710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:43.107433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-61b50316-cdda-4265-9d83-8c7f885a6c82 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e3d973693cc45d0abaa391442971e88', 'request_id': 'req-61b50316-cdda-4265-9d83-8c7f885a6c82'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:43.132512 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-61b50316-cdda-4265-9d83-8c7f885a6c82 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e3d973693cc45d0abaa391442971e88', 'request_id': 'req-61b50316-cdda-4265-9d83-8c7f885a6c82'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:43.168763 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-61b50316-cdda-4265-9d83-8c7f885a6c82 admin admin] The token cache is being invalidate because project 2e3d973693cc45d0abaa391442971e88 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:30:43.168763 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-61b50316-cdda-4265-9d83-8c7f885a6c82 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:43.170499 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-61b50316-cdda-4265-9d83-8c7f885a6c82 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:43.175501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 925/1853] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:30:42 2022] DELETE /identity/v3/projects/2e3d973693cc45d0abaa391442971e88 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:43.180867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e073698-d793-4fdf-9f9a-6c75af2ff21a service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:43.216548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e073698-d793-4fdf-9f9a-6c75af2ff21a service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:43.389140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec93f6d6-4201-4b86-b108-c322c2295089 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:43.389975 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec93f6d6-4201-4b86-b108-c322c2295089 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:43.390667 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec93f6d6-4201-4b86-b108-c322c2295089 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:43.392293 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec93f6d6-4201-4b86-b108-c322c2295089 admin admin] PATH_INFO: `/v3/users/b9cdd4d2faf248cc85533f990a47fa7c` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:43.400434 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec93f6d6-4201-4b86-b108-c322c2295089 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b9cdd4d2faf248cc85533f990a47fa7c)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:43.403451 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec93f6d6-4201-4b86-b108-c322c2295089 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:43.525540 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ec93f6d6-4201-4b86-b108-c322c2295089 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b9cdd4d2faf248cc85533f990a47fa7c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:43.543247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ec93f6d6-4201-4b86-b108-c322c2295089 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b9cdd4d2faf248cc85533f990a47fa7c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:43.567645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ec93f6d6-4201-4b86-b108-c322c2295089 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b9cdd4d2faf248cc85533f990a47fa7c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:43.593957 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:43.601174 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ec93f6d6-4201-4b86-b108-c322c2295089 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b9cdd4d2faf248cc85533f990a47fa7c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:43.602532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ec93f6d6-4201-4b86-b108-c322c2295089 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b9cdd4d2faf248cc85533f990a47fa7c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:43.612295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 929/1854] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:30:43 2022] DELETE /identity/v3/users/b9cdd4d2faf248cc85533f990a47fa7c => generated 0 bytes in 432 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:44.183722 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b50316-cdda-4265-9d83-8c7f885a6c82 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:44.185319 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-61b50316-cdda-4265-9d83-8c7f885a6c82 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:44.205394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b50316-cdda-4265-9d83-8c7f885a6c82 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:44.313653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193c4440-0415-445e-ba6a-77e294cd3dce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:44.314894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193c4440-0415-445e-ba6a-77e294cd3dce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:44.315561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193c4440-0415-445e-ba6a-77e294cd3dce admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:44.315794 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193c4440-0415-445e-ba6a-77e294cd3dce admin admin] PATH_INFO: `/v3/projects/6841f8c226564d6ab5ff4ac6df9c89c7` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:44.317107 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193c4440-0415-445e-ba6a-77e294cd3dce admin admin] RBAC: Authorizing `identity:delete_project(project_id=6841f8c226564d6ab5ff4ac6df9c89c7)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:44.321344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193c4440-0415-445e-ba6a-77e294cd3dce admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:44.376230 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-193c4440-0415-445e-ba6a-77e294cd3dce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6841f8c226564d6ab5ff4ac6df9c89c7', 'request_id': 'req-193c4440-0415-445e-ba6a-77e294cd3dce'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:44.385656 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-193c4440-0415-445e-ba6a-77e294cd3dce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6841f8c226564d6ab5ff4ac6df9c89c7', 'request_id': 'req-193c4440-0415-445e-ba6a-77e294cd3dce'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:44.392580 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-193c4440-0415-445e-ba6a-77e294cd3dce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6841f8c226564d6ab5ff4ac6df9c89c7', 'request_id': 'req-193c4440-0415-445e-ba6a-77e294cd3dce'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:44.410237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-193c4440-0415-445e-ba6a-77e294cd3dce admin admin] The token cache is being invalidate because project 6841f8c226564d6ab5ff4ac6df9c89c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:30:44.410541 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-193c4440-0415-445e-ba6a-77e294cd3dce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:44.411262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-193c4440-0415-445e-ba6a-77e294cd3dce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:44.413408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 926/1855] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:30:44 2022] DELETE /identity/v3/projects/6841f8c226564d6ab5ff4ac6df9c89c7 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:44.418580 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc22f952-6c9b-4394-860d-bd61bace58e9 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:44.418867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc22f952-6c9b-4394-860d-bd61bace58e9 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:44.419143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc22f952-6c9b-4394-860d-bd61bace58e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:44.439984 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-cc22f952-6c9b-4394-860d-bd61bace58e9 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:44.441525 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-cc22f952-6c9b-4394-860d-bd61bace58e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:44.541095 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 930/1856] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:44.542629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe05648-7c77-465a-947f-d5c45fba498d None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:44.543075 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe05648-7c77-465a-947f-d5c45fba498d None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:44.543437 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe05648-7c77-465a-947f-d5c45fba498d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:44.569002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-4fe05648-7c77-465a-947f-d5c45fba498d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:44.571114 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-4fe05648-7c77-465a-947f-d5c45fba498d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:44.583886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 927/1857] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:44.585214 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc22f952-6c9b-4394-860d-bd61bace58e9 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:44.609763 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc22f952-6c9b-4394-860d-bd61bace58e9 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:44.618051 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b64c8c2-1438-4d1c-a008-83bb25d29ec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:44.618783 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b64c8c2-1438-4d1c-a008-83bb25d29ec0 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:44.619295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b64c8c2-1438-4d1c-a008-83bb25d29ec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:44.619666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b64c8c2-1438-4d1c-a008-83bb25d29ec0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:44.620587 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b64c8c2-1438-4d1c-a008-83bb25d29ec0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:30:44.621073 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b64c8c2-1438-4d1c-a008-83bb25d29ec0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:44.623433 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b64c8c2-1438-4d1c-a008-83bb25d29ec0 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:44.640106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 931/1858] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:30:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:30:44.645287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe05648-7c77-465a-947f-d5c45fba498d None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:44.658544 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe05648-7c77-465a-947f-d5c45fba498d None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:44.671263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6f0d9a-1cca-4f43-acba-d92ff35fa0b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:44.672381 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6f0d9a-1cca-4f43-acba-d92ff35fa0b6 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:44.672813 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6f0d9a-1cca-4f43-acba-d92ff35fa0b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:44.673390 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6f0d9a-1cca-4f43-acba-d92ff35fa0b6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:44.674064 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6f0d9a-1cca-4f43-acba-d92ff35fa0b6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:44.677441 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6f0d9a-1cca-4f43-acba-d92ff35fa0b6 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:44.712946 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 928/1859] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:30:44 2022] POST /identity/v3/projects => generated 401 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:44.717093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b64c8c2-1438-4d1c-a008-83bb25d29ec0 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:44.728712 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b64c8c2-1438-4d1c-a008-83bb25d29ec0 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:44.850307 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19265da6-3f6b-4e19-bc89-46889ba5225f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:44.851001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19265da6-3f6b-4e19-bc89-46889ba5225f admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:44.851232 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19265da6-3f6b-4e19-bc89-46889ba5225f admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:44.851948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19265da6-3f6b-4e19-bc89-46889ba5225f admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:44.852145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19265da6-3f6b-4e19-bc89-46889ba5225f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:44.854168 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19265da6-3f6b-4e19-bc89-46889ba5225f admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:44.883412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 932/1860] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:30:44 2022] POST /identity/v3/users => generated 301 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:30:44.887439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6f0d9a-1cca-4f43-acba-d92ff35fa0b6 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:44.901196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6f0d9a-1cca-4f43-acba-d92ff35fa0b6 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:44.909359 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1688b8f-532c-4df7-9033-8c95669cc640 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:44.910112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1688b8f-532c-4df7-9033-8c95669cc640 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:44.910939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1688b8f-532c-4df7-9033-8c95669cc640 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:44.910939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1688b8f-532c-4df7-9033-8c95669cc640 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:44.912253 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1688b8f-532c-4df7-9033-8c95669cc640 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:44.914047 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1688b8f-532c-4df7-9033-8c95669cc640 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:44.929736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 929/1861] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:44 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:44.935236 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19265da6-3f6b-4e19-bc89-46889ba5225f admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:44.953322 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19265da6-3f6b-4e19-bc89-46889ba5225f admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:44.962122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05dfd38-a2fa-42f2-aae5-3f4cd0def19f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:44.963095 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05dfd38-a2fa-42f2-aae5-3f4cd0def19f admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:44.963822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05dfd38-a2fa-42f2-aae5-3f4cd0def19f admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:44.964342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05dfd38-a2fa-42f2-aae5-3f4cd0def19f admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:44.965176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05dfd38-a2fa-42f2-aae5-3f4cd0def19f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:44.967084 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05dfd38-a2fa-42f2-aae5-3f4cd0def19f admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:44.982214 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 933/1862] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:44 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:44.988190 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1688b8f-532c-4df7-9033-8c95669cc640 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:45.021257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1688b8f-532c-4df7-9033-8c95669cc640 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:45.021257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f587a504-a4f9-4e5f-a0b6-ce6b836e4941 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:45.021257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f587a504-a4f9-4e5f-a0b6-ce6b836e4941 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:45.021257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f587a504-a4f9-4e5f-a0b6-ce6b836e4941 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:45.021910 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f587a504-a4f9-4e5f-a0b6-ce6b836e4941 admin admin] PATH_INFO: `/v3/projects/fa08528a9a514fcbba774ee47e8fa6d5/users/3d7aab6eb35f4aa5a8685db2ae0d28c3/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:45.046597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f587a504-a4f9-4e5f-a0b6-ce6b836e4941 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa08528a9a514fcbba774ee47e8fa6d5, user_id=3d7aab6eb35f4aa5a8685db2ae0d28c3, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:45.050471 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f587a504-a4f9-4e5f-a0b6-ce6b836e4941 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:45.071469 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 930/1863] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:44 2022] PUT /identity/v3/projects/fa08528a9a514fcbba774ee47e8fa6d5/users/3d7aab6eb35f4aa5a8685db2ae0d28c3/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:45.081785 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e89222-4311-40a3-bbd2-b495a1ec2673 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:45.082333 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e89222-4311-40a3-bbd2-b495a1ec2673 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:45.082820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e89222-4311-40a3-bbd2-b495a1ec2673 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:45.127197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-51e89222-4311-40a3-bbd2-b495a1ec2673 None None] MFA Rules not processed for user `3d7aab6eb35f4aa5a8685db2ae0d28c3`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:45.128451 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-51e89222-4311-40a3-bbd2-b495a1ec2673 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:45.201170 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:DELETE=1,keystone:SELECT=96,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:45.230698 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 934/1864] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:45 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:30:45.235269 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f587a504-a4f9-4e5f-a0b6-ce6b836e4941 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:45.242387 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:45.328972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f587a504-a4f9-4e5f-a0b6-ce6b836e4941 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:45.336643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9b74f9-7e5b-4ddb-b834-87c6bf52265c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:45.337454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9b74f9-7e5b-4ddb-b834-87c6bf52265c admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:45.337834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9b74f9-7e5b-4ddb-b834-87c6bf52265c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:45.338209 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9b74f9-7e5b-4ddb-b834-87c6bf52265c admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:45.339054 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9b74f9-7e5b-4ddb-b834-87c6bf52265c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:45.341261 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9b74f9-7e5b-4ddb-b834-87c6bf52265c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:45.381204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 931/1865] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:30:45 2022] POST /identity/v3/projects => generated 401 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:45.385196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e89222-4311-40a3-bbd2-b495a1ec2673 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:45.407992 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e89222-4311-40a3-bbd2-b495a1ec2673 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:45.487766 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff0cdff-ef47-4431-9771-edf0910cc706 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:45.488432 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff0cdff-ef47-4431-9771-edf0910cc706 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:45.488718 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff0cdff-ef47-4431-9771-edf0910cc706 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:45.488988 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff0cdff-ef47-4431-9771-edf0910cc706 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:45.490559 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff0cdff-ef47-4431-9771-edf0910cc706 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:45.492673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff0cdff-ef47-4431-9771-edf0910cc706 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:45.520132 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 935/1866] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:30:45 2022] POST /identity/v3/users => generated 307 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:30:45.521954 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9b74f9-7e5b-4ddb-b834-87c6bf52265c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:45.539870 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9b74f9-7e5b-4ddb-b834-87c6bf52265c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:45.551271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa19c36f-a87b-46bf-ab60-522f6f847862 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:45.552937 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa19c36f-a87b-46bf-ab60-522f6f847862 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:45.553394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa19c36f-a87b-46bf-ab60-522f6f847862 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:45.554502 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa19c36f-a87b-46bf-ab60-522f6f847862 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:45.555443 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa19c36f-a87b-46bf-ab60-522f6f847862 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:45.557243 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa19c36f-a87b-46bf-ab60-522f6f847862 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:45.573234 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 932/1867] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:45 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:45.575787 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff0cdff-ef47-4431-9771-edf0910cc706 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:45.587745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff0cdff-ef47-4431-9771-edf0910cc706 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:45.593569 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29c6ede-3ff5-4181-9961-0be1b7d9654c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:45.594551 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29c6ede-3ff5-4181-9961-0be1b7d9654c admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:45.594938 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29c6ede-3ff5-4181-9961-0be1b7d9654c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:45.595317 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29c6ede-3ff5-4181-9961-0be1b7d9654c admin admin] PATH_INFO: `/v3/projects/2ab957150d7c4a269fbf4924e0606c2b/users/20db61fcda1849daafa0b83aaa531324/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:45.618524 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29c6ede-3ff5-4181-9961-0be1b7d9654c admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ab957150d7c4a269fbf4924e0606c2b, user_id=20db61fcda1849daafa0b83aaa531324, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:45.621584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29c6ede-3ff5-4181-9961-0be1b7d9654c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:45.681146 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 936/1868] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:45 2022] PUT /identity/v3/projects/2ab957150d7c4a269fbf4924e0606c2b/users/20db61fcda1849daafa0b83aaa531324/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:45.684986 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa19c36f-a87b-46bf-ab60-522f6f847862 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:45.701169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa19c36f-a87b-46bf-ab60-522f6f847862 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:45.796395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf51580-cba3-4d15-8549-cee5776d29bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:45.797280 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf51580-cba3-4d15-8549-cee5776d29bc admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:45.797655 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf51580-cba3-4d15-8549-cee5776d29bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:45.798007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf51580-cba3-4d15-8549-cee5776d29bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:45.799435 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf51580-cba3-4d15-8549-cee5776d29bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:45.801551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf51580-cba3-4d15-8549-cee5776d29bc admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:45.818623 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 933/1869] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:45 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:45.822871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29c6ede-3ff5-4181-9961-0be1b7d9654c admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:45.844214 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29c6ede-3ff5-4181-9961-0be1b7d9654c admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:45.862575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af60824b-40c5-4b05-b3d4-4d4c1bebf5a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:45.864080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af60824b-40c5-4b05-b3d4-4d4c1bebf5a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:45.865279 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af60824b-40c5-4b05-b3d4-4d4c1bebf5a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:45.865714 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af60824b-40c5-4b05-b3d4-4d4c1bebf5a9 admin admin] PATH_INFO: `/v3/projects/2ab957150d7c4a269fbf4924e0606c2b/users/20db61fcda1849daafa0b83aaa531324/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:45.868099 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af60824b-40c5-4b05-b3d4-4d4c1bebf5a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ab957150d7c4a269fbf4924e0606c2b, user_id=20db61fcda1849daafa0b83aaa531324, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:45.871424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af60824b-40c5-4b05-b3d4-4d4c1bebf5a9 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:45.891832 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 937/1870] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:45 2022] PUT /identity/v3/projects/2ab957150d7c4a269fbf4924e0606c2b/users/20db61fcda1849daafa0b83aaa531324/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:45.904472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74730fc-0a77-4d59-addd-292d338a47f1 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:45.905201 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74730fc-0a77-4d59-addd-292d338a47f1 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:45.905487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74730fc-0a77-4d59-addd-292d338a47f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:45.948142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-b74730fc-0a77-4d59-addd-292d338a47f1 None None] MFA Rules not processed for user `20db61fcda1849daafa0b83aaa531324`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:45.949405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-b74730fc-0a77-4d59-addd-292d338a47f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:46.048257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:46.053323 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 934/1871] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:45 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:46.066285 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af60824b-40c5-4b05-b3d4-4d4c1bebf5a9 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:46.068692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-af60824b-40c5-4b05-b3d4-4d4c1bebf5a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:46.214765 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af60824b-40c5-4b05-b3d4-4d4c1bebf5a9 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:46.230443 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932255e8-a21c-43f5-b050-60838b74024b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:46.231175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932255e8-a21c-43f5-b050-60838b74024b service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:46.231315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932255e8-a21c-43f5-b050-60838b74024b service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:46.231584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932255e8-a21c-43f5-b050-60838b74024b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:46.265177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932255e8-a21c-43f5-b050-60838b74024b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:46.267034 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932255e8-a21c-43f5-b050-60838b74024b service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:46.285516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 938/1872] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:46 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:46.376426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74730fc-0a77-4d59-addd-292d338a47f1 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:46.393756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74730fc-0a77-4d59-addd-292d338a47f1 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:46.400996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe4f5bc-22f3-4356-b0b2-0af6ce9f56a4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:46.400996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe4f5bc-22f3-4356-b0b2-0af6ce9f56a4 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:46.400996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe4f5bc-22f3-4356-b0b2-0af6ce9f56a4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:46.400996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe4f5bc-22f3-4356-b0b2-0af6ce9f56a4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:46.487707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe4f5bc-22f3-4356-b0b2-0af6ce9f56a4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:46.489210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe4f5bc-22f3-4356-b0b2-0af6ce9f56a4 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:46.500813 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 935/1873] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:46 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:49.486563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932255e8-a21c-43f5-b050-60838b74024b service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:49.488752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-932255e8-a21c-43f5-b050-60838b74024b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:49.616151 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:49.647382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932255e8-a21c-43f5-b050-60838b74024b service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:49.673489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2785fb0e-7756-467a-9721-2b4346ef2603 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:49.673489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2785fb0e-7756-467a-9721-2b4346ef2603 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:49.673489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2785fb0e-7756-467a-9721-2b4346ef2603 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:49.673489 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2785fb0e-7756-467a-9721-2b4346ef2603 admin admin] PATH_INFO: `/v3/users/78520885bf274596af992937a9c1a397` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:49.673843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2785fb0e-7756-467a-9721-2b4346ef2603 admin admin] RBAC: Authorizing `identity:delete_user(user_id=78520885bf274596af992937a9c1a397)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:49.674173 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2785fb0e-7756-467a-9721-2b4346ef2603 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:49.768004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-2785fb0e-7756-467a-9721-2b4346ef2603 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '78520885bf274596af992937a9c1a397'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:49.779804 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-2785fb0e-7756-467a-9721-2b4346ef2603 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '78520885bf274596af992937a9c1a397'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:49.794985 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-2785fb0e-7756-467a-9721-2b4346ef2603 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '78520885bf274596af992937a9c1a397'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:49.820421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-2785fb0e-7756-467a-9721-2b4346ef2603 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '78520885bf274596af992937a9c1a397'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:49.821691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-2785fb0e-7756-467a-9721-2b4346ef2603 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '78520885bf274596af992937a9c1a397'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:49.833114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 939/1874] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:30:49 2022] DELETE /identity/v3/users/78520885bf274596af992937a9c1a397 => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:49.838331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afe1e36-c042-4651-97d8-7e4c29d4e975 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:49.839274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afe1e36-c042-4651-97d8-7e4c29d4e975 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:49.839274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afe1e36-c042-4651-97d8-7e4c29d4e975 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:49.863911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-4afe1e36-c042-4651-97d8-7e4c29d4e975 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:49.865434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-4afe1e36-c042-4651-97d8-7e4c29d4e975 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:49.938035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 936/1875] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:49.949712 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2785fb0e-7756-467a-9721-2b4346ef2603 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:49.951648 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-2785fb0e-7756-467a-9721-2b4346ef2603 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:50.023289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2785fb0e-7756-467a-9721-2b4346ef2603 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:50.028262 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900e3f12-f829-4585-b342-aba69fa4fe6a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:50.028928 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900e3f12-f829-4585-b342-aba69fa4fe6a service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:50.029242 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900e3f12-f829-4585-b342-aba69fa4fe6a service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:50.029476 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900e3f12-f829-4585-b342-aba69fa4fe6a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:50.043394 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900e3f12-f829-4585-b342-aba69fa4fe6a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:50.044891 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900e3f12-f829-4585-b342-aba69fa4fe6a service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:50.056517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 940/1876] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:50.725106 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afe1e36-c042-4651-97d8-7e4c29d4e975 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:50.726871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-4afe1e36-c042-4651-97d8-7e4c29d4e975 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:50.746975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afe1e36-c042-4651-97d8-7e4c29d4e975 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:50.757459 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb297d7-2635-402a-8f6e-68a64803bc6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:50.761165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb297d7-2635-402a-8f6e-68a64803bc6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:50.761907 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb297d7-2635-402a-8f6e-68a64803bc6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:50.762707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb297d7-2635-402a-8f6e-68a64803bc6a admin admin] PATH_INFO: `/v3/projects/b513e7d714ca432b8e9e83fe9be608f1` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:50.764372 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb297d7-2635-402a-8f6e-68a64803bc6a admin admin] RBAC: Authorizing `identity:delete_project(project_id=b513e7d714ca432b8e9e83fe9be608f1)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:50.767866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb297d7-2635-402a-8f6e-68a64803bc6a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:50.872662 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-efb297d7-2635-402a-8f6e-68a64803bc6a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b513e7d714ca432b8e9e83fe9be608f1', 'request_id': 'req-efb297d7-2635-402a-8f6e-68a64803bc6a'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:50.884254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-efb297d7-2635-402a-8f6e-68a64803bc6a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b513e7d714ca432b8e9e83fe9be608f1', 'request_id': 'req-efb297d7-2635-402a-8f6e-68a64803bc6a'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:50.896967 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-efb297d7-2635-402a-8f6e-68a64803bc6a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b513e7d714ca432b8e9e83fe9be608f1', 'request_id': 'req-efb297d7-2635-402a-8f6e-68a64803bc6a'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:50.922274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-efb297d7-2635-402a-8f6e-68a64803bc6a admin admin] The token cache is being invalidate because project b513e7d714ca432b8e9e83fe9be608f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:30:50.922657 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-efb297d7-2635-402a-8f6e-68a64803bc6a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:50.923628 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-efb297d7-2635-402a-8f6e-68a64803bc6a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:50.928823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 937/1877] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:30:50 2022] DELETE /identity/v3/projects/b513e7d714ca432b8e9e83fe9be608f1 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:30:50.929986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900e3f12-f829-4585-b342-aba69fa4fe6a service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:50.947667 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900e3f12-f829-4585-b342-aba69fa4fe6a service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:51.079329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a902adb3-ea54-43c7-a578-b78a3072da83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:51.079994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a902adb3-ea54-43c7-a578-b78a3072da83 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:51.080283 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a902adb3-ea54-43c7-a578-b78a3072da83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:51.080600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a902adb3-ea54-43c7-a578-b78a3072da83 admin admin] PATH_INFO: `/v3/users/21f47f5bd8b440f49e60d49f7a7b726d` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:51.081848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a902adb3-ea54-43c7-a578-b78a3072da83 admin admin] RBAC: Authorizing `identity:delete_user(user_id=21f47f5bd8b440f49e60d49f7a7b726d)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:51.085385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a902adb3-ea54-43c7-a578-b78a3072da83 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:51.092794 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:51.280286 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-a902adb3-ea54-43c7-a578-b78a3072da83 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21f47f5bd8b440f49e60d49f7a7b726d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:51.292002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-a902adb3-ea54-43c7-a578-b78a3072da83 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21f47f5bd8b440f49e60d49f7a7b726d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:51.302103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-a902adb3-ea54-43c7-a578-b78a3072da83 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21f47f5bd8b440f49e60d49f7a7b726d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:51.321636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-a902adb3-ea54-43c7-a578-b78a3072da83 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21f47f5bd8b440f49e60d49f7a7b726d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:51.322372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-a902adb3-ea54-43c7-a578-b78a3072da83 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21f47f5bd8b440f49e60d49f7a7b726d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:51.331383 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 941/1878] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:30:50 2022] DELETE /identity/v3/users/21f47f5bd8b440f49e60d49f7a7b726d => generated 0 bytes in 402 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:30:52.100072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb297d7-2635-402a-8f6e-68a64803bc6a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:52.103100 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-efb297d7-2635-402a-8f6e-68a64803bc6a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:52.127868 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb297d7-2635-402a-8f6e-68a64803bc6a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:52.171801 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:52.217622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9283188-9863-4704-86a8-46e407183f1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:52.218267 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9283188-9863-4704-86a8-46e407183f1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:52.218549 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9283188-9863-4704-86a8-46e407183f1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:52.218824 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9283188-9863-4704-86a8-46e407183f1b admin admin] PATH_INFO: `/v3/projects/7ab22ef2ab16440cb2d08e6d141e14ed` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:52.219999 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9283188-9863-4704-86a8-46e407183f1b admin admin] RBAC: Authorizing `identity:delete_project(project_id=7ab22ef2ab16440cb2d08e6d141e14ed)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:52.222036 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9283188-9863-4704-86a8-46e407183f1b admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:52.287820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c9283188-9863-4704-86a8-46e407183f1b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7ab22ef2ab16440cb2d08e6d141e14ed', 'request_id': 'req-c9283188-9863-4704-86a8-46e407183f1b'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:52.299717 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c9283188-9863-4704-86a8-46e407183f1b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7ab22ef2ab16440cb2d08e6d141e14ed', 'request_id': 'req-c9283188-9863-4704-86a8-46e407183f1b'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:52.310111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c9283188-9863-4704-86a8-46e407183f1b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7ab22ef2ab16440cb2d08e6d141e14ed', 'request_id': 'req-c9283188-9863-4704-86a8-46e407183f1b'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:52.332190 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c9283188-9863-4704-86a8-46e407183f1b admin admin] The token cache is being invalidate because project 7ab22ef2ab16440cb2d08e6d141e14ed was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:30:52.332630 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c9283188-9863-4704-86a8-46e407183f1b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:52.334081 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c9283188-9863-4704-86a8-46e407183f1b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:52.335977 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 938/1879] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:30:52 2022] DELETE /identity/v3/projects/7ab22ef2ab16440cb2d08e6d141e14ed => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:52.342061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b461075-9177-4561-96c1-f70bf6a2ec02 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:52.342571 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b461075-9177-4561-96c1-f70bf6a2ec02 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:52.342648 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b461075-9177-4561-96c1-f70bf6a2ec02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:52.368596 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-3b461075-9177-4561-96c1-f70bf6a2ec02 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:52.370048 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-3b461075-9177-4561-96c1-f70bf6a2ec02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:52.492397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 942/1880] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:30:52.499940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce4b1e3-d06e-4b10-83b9-fd9a86f43dc7 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:52.500227 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce4b1e3-d06e-4b10-83b9-fd9a86f43dc7 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:52.500550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce4b1e3-d06e-4b10-83b9-fd9a86f43dc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:52.543551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-1ce4b1e3-d06e-4b10-83b9-fd9a86f43dc7 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:52.545568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-1ce4b1e3-d06e-4b10-83b9-fd9a86f43dc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:52.559762 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 939/1881] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:52.560634 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b461075-9177-4561-96c1-f70bf6a2ec02 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:52.571891 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b461075-9177-4561-96c1-f70bf6a2ec02 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:52.578195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b899c12-6d7b-487b-b7d7-a62a9634e7b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:52.579117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b899c12-6d7b-487b-b7d7-a62a9634e7b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:52.579885 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b899c12-6d7b-487b-b7d7-a62a9634e7b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:52.580415 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b899c12-6d7b-487b-b7d7-a62a9634e7b8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:52.582045 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b899c12-6d7b-487b-b7d7-a62a9634e7b8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:30:52.582572 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b899c12-6d7b-487b-b7d7-a62a9634e7b8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:52.585582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b899c12-6d7b-487b-b7d7-a62a9634e7b8 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:52.603076 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 943/1882] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:30:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:30:52.607533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce4b1e3-d06e-4b10-83b9-fd9a86f43dc7 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:52.619017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce4b1e3-d06e-4b10-83b9-fd9a86f43dc7 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:52.624441 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4c42b6-2058-4ed0-a57b-f1a854586b37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:52.625063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4c42b6-2058-4ed0-a57b-f1a854586b37 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:52.625357 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4c42b6-2058-4ed0-a57b-f1a854586b37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:52.625615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4c42b6-2058-4ed0-a57b-f1a854586b37 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:52.626257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4c42b6-2058-4ed0-a57b-f1a854586b37 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:52.628169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4c42b6-2058-4ed0-a57b-f1a854586b37 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:52.658988 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b899c12-6d7b-487b-b7d7-a62a9634e7b8 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:52.663593 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-2b899c12-6d7b-487b-b7d7-a62a9634e7b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:52.670432 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 940/1883] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:30:52 2022] POST /identity/v3/projects => generated 371 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:52.672483 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4c42b6-2058-4ed0-a57b-f1a854586b37 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:52.682358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b899c12-6d7b-487b-b7d7-a62a9634e7b8 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:52.776655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1c2020-efca-44f2-8316-fed31fcaa032 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:52.778344 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1c2020-efca-44f2-8316-fed31fcaa032 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:52.778728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1c2020-efca-44f2-8316-fed31fcaa032 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:52.779104 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1c2020-efca-44f2-8316-fed31fcaa032 admin admin] PATH_INFO: `/v3/users/3d7aab6eb35f4aa5a8685db2ae0d28c3` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:52.780657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1c2020-efca-44f2-8316-fed31fcaa032 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3d7aab6eb35f4aa5a8685db2ae0d28c3)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:52.783334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1c2020-efca-44f2-8316-fed31fcaa032 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:52.806950 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:52.831678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4c42b6-2058-4ed0-a57b-f1a854586b37 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:52.838171 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef9799e-54b6-44ea-a62e-36779af3227c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:52.838989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef9799e-54b6-44ea-a62e-36779af3227c admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:52.839276 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef9799e-54b6-44ea-a62e-36779af3227c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:52.839550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef9799e-54b6-44ea-a62e-36779af3227c admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:52.840210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef9799e-54b6-44ea-a62e-36779af3227c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:52.842509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef9799e-54b6-44ea-a62e-36779af3227c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:52.869526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bf1c2020-efca-44f2-8316-fed31fcaa032 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3d7aab6eb35f4aa5a8685db2ae0d28c3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:52.883935 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bf1c2020-efca-44f2-8316-fed31fcaa032 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3d7aab6eb35f4aa5a8685db2ae0d28c3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:52.889385 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 941/1884] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:30:52 2022] POST /identity/v3/users => generated 286 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:52.890296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef9799e-54b6-44ea-a62e-36779af3227c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:52.907871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bf1c2020-efca-44f2-8316-fed31fcaa032 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3d7aab6eb35f4aa5a8685db2ae0d28c3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:52.913095 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef9799e-54b6-44ea-a62e-36779af3227c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:52.920762 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1853fb-dc6b-4889-a362-37a00b48388d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:52.922564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1853fb-dc6b-4889-a362-37a00b48388d admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:52.922564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1853fb-dc6b-4889-a362-37a00b48388d admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:52.922564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1853fb-dc6b-4889-a362-37a00b48388d admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:52.922564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1853fb-dc6b-4889-a362-37a00b48388d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:52.924662 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1853fb-dc6b-4889-a362-37a00b48388d admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:52.944390 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 942/1885] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:52 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:52.949094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1853fb-dc6b-4889-a362-37a00b48388d admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:52.950858 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bf1c2020-efca-44f2-8316-fed31fcaa032 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3d7aab6eb35f4aa5a8685db2ae0d28c3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:52.952952 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bf1c2020-efca-44f2-8316-fed31fcaa032 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3d7aab6eb35f4aa5a8685db2ae0d28c3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:52.958696 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:52.973461 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 944/1886] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:30:52 2022] DELETE /identity/v3/users/3d7aab6eb35f4aa5a8685db2ae0d28c3 => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:52.978388 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3af10fd-7fe8-4451-b2a4-6672fe3454c8 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:52.978706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3af10fd-7fe8-4451-b2a4-6672fe3454c8 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:52.978807 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1853fb-dc6b-4889-a362-37a00b48388d admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:52.978966 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3af10fd-7fe8-4451-b2a4-6672fe3454c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:53.000179 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-f3af10fd-7fe8-4451-b2a4-6672fe3454c8 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:53.001458 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f3af10fd-7fe8-4451-b2a4-6672fe3454c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:53.091728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 945/1887] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:53.093699 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32d6caa-59b5-48f1-84dc-e64ce49d07fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:53.094471 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32d6caa-59b5-48f1-84dc-e64ce49d07fc admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:53.095631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32d6caa-59b5-48f1-84dc-e64ce49d07fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:53.095631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32d6caa-59b5-48f1-84dc-e64ce49d07fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:53.096512 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32d6caa-59b5-48f1-84dc-e64ce49d07fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:53.099682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32d6caa-59b5-48f1-84dc-e64ce49d07fc admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:53.104121 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3af10fd-7fe8-4451-b2a4-6672fe3454c8 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:53.107136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f3af10fd-7fe8-4451-b2a4-6672fe3454c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:53.117431 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 943/1888] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:52 2022] GET /identity/v3/roles => generated 1420 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:53.123487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32d6caa-59b5-48f1-84dc-e64ce49d07fc admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:53.126448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a32d6caa-59b5-48f1-84dc-e64ce49d07fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:53.149334 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32d6caa-59b5-48f1-84dc-e64ce49d07fc admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:53.164634 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491e1e37-972d-40eb-92d7-130c2efd37cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:53.165377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491e1e37-972d-40eb-92d7-130c2efd37cc admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:53.165657 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491e1e37-972d-40eb-92d7-130c2efd37cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:53.165937 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491e1e37-972d-40eb-92d7-130c2efd37cc admin admin] PATH_INFO: `/v3/projects/ff877a220f9443fb93fde985a653c999/users/b3bd51e1acb548ea957ffb57cb09f3f4/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:53.209750 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491e1e37-972d-40eb-92d7-130c2efd37cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff877a220f9443fb93fde985a653c999, user_id=b3bd51e1acb548ea957ffb57cb09f3f4, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:53.212393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3af10fd-7fe8-4451-b2a4-6672fe3454c8 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:53.215276 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491e1e37-972d-40eb-92d7-130c2efd37cc admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:53.240569 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 944/1889] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:53 2022] PUT /identity/v3/projects/ff877a220f9443fb93fde985a653c999/users/b3bd51e1acb548ea957ffb57cb09f3f4/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:53.249167 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63738698-75f0-4109-8eb2-5e5211195521 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:53.249570 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63738698-75f0-4109-8eb2-5e5211195521 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:53.250258 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddea4db-26a1-44c3-baf8-1f6d7a6b4899 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:53.250918 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddea4db-26a1-44c3-baf8-1f6d7a6b4899 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:53.251338 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddea4db-26a1-44c3-baf8-1f6d7a6b4899 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:53.251595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddea4db-26a1-44c3-baf8-1f6d7a6b4899 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:53.254546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63738698-75f0-4109-8eb2-5e5211195521 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:53.264059 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddea4db-26a1-44c3-baf8-1f6d7a6b4899 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:53.265741 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddea4db-26a1-44c3-baf8-1f6d7a6b4899 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:53.283194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-63738698-75f0-4109-8eb2-5e5211195521 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:53.284514 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-63738698-75f0-4109-8eb2-5e5211195521 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:53.285385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 946/1890] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:30:53.389542 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 945/1891] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:53.404122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddea4db-26a1-44c3-baf8-1f6d7a6b4899 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:53.407277 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6ddea4db-26a1-44c3-baf8-1f6d7a6b4899 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:53.427281 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddea4db-26a1-44c3-baf8-1f6d7a6b4899 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:53.484270 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:53.511252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a82380-b8c5-41de-9113-aea696588d2d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:53.511891 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a82380-b8c5-41de-9113-aea696588d2d service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:53.512217 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a82380-b8c5-41de-9113-aea696588d2d service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:53.512570 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a82380-b8c5-41de-9113-aea696588d2d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:53.525047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a82380-b8c5-41de-9113-aea696588d2d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:53.526838 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a82380-b8c5-41de-9113-aea696588d2d service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:53.543657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 947/1892] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:53.752582 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63738698-75f0-4109-8eb2-5e5211195521 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:53.754317 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-63738698-75f0-4109-8eb2-5e5211195521 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:53.793449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63738698-75f0-4109-8eb2-5e5211195521 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:53.814491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5c3859-c2c9-4851-b546-0c806e377a3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:53.816644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5c3859-c2c9-4851-b546-0c806e377a3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:53.816915 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5c3859-c2c9-4851-b546-0c806e377a3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:53.817249 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5c3859-c2c9-4851-b546-0c806e377a3c admin admin] PATH_INFO: `/v3/projects/fa08528a9a514fcbba774ee47e8fa6d5` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:53.822432 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5c3859-c2c9-4851-b546-0c806e377a3c admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa08528a9a514fcbba774ee47e8fa6d5)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:53.836582 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5c3859-c2c9-4851-b546-0c806e377a3c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:53.995244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8b5c3859-c2c9-4851-b546-0c806e377a3c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa08528a9a514fcbba774ee47e8fa6d5', 'request_id': 'req-8b5c3859-c2c9-4851-b546-0c806e377a3c'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:54.010438 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8b5c3859-c2c9-4851-b546-0c806e377a3c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa08528a9a514fcbba774ee47e8fa6d5', 'request_id': 'req-8b5c3859-c2c9-4851-b546-0c806e377a3c'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:54.019682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8b5c3859-c2c9-4851-b546-0c806e377a3c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa08528a9a514fcbba774ee47e8fa6d5', 'request_id': 'req-8b5c3859-c2c9-4851-b546-0c806e377a3c'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:54.027004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a82380-b8c5-41de-9113-aea696588d2d service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:54.028703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-67a82380-b8c5-41de-9113-aea696588d2d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:54.039463 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:54.048259 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8b5c3859-c2c9-4851-b546-0c806e377a3c admin admin] The token cache is being invalidate because project fa08528a9a514fcbba774ee47e8fa6d5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:30:54.048985 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8b5c3859-c2c9-4851-b546-0c806e377a3c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:54.052382 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8b5c3859-c2c9-4851-b546-0c806e377a3c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:54.054893 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 946/1893] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:30:53 2022] DELETE /identity/v3/projects/fa08528a9a514fcbba774ee47e8fa6d5 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:54.058804 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5c3859-c2c9-4851-b546-0c806e377a3c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:54.060096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a82380-b8c5-41de-9113-aea696588d2d service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:54.061099 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8b5c3859-c2c9-4851-b546-0c806e377a3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:54.193607 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5c3859-c2c9-4851-b546-0c806e377a3c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:54.206058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6a7a06-d588-4df2-b148-bd5fedcada16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:54.207047 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6a7a06-d588-4df2-b148-bd5fedcada16 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:54.208809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6a7a06-d588-4df2-b148-bd5fedcada16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:54.209354 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6a7a06-d588-4df2-b148-bd5fedcada16 admin admin] PATH_INFO: `/v3/users/20db61fcda1849daafa0b83aaa531324` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:54.211495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6a7a06-d588-4df2-b148-bd5fedcada16 admin admin] RBAC: Authorizing `identity:delete_user(user_id=20db61fcda1849daafa0b83aaa531324)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:54.216096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6a7a06-d588-4df2-b148-bd5fedcada16 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:54.231240 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd105fe-18dd-4252-b8ec-b57a1a8cd0e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:54.231907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd105fe-18dd-4252-b8ec-b57a1a8cd0e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:54.232201 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd105fe-18dd-4252-b8ec-b57a1a8cd0e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:54.232594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd105fe-18dd-4252-b8ec-b57a1a8cd0e0 admin admin] PATH_INFO: `/v3/users/eacb5c92a3574921b503d6ead653a66d` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:54.235493 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd105fe-18dd-4252-b8ec-b57a1a8cd0e0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eacb5c92a3574921b503d6ead653a66d)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:54.237650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd105fe-18dd-4252-b8ec-b57a1a8cd0e0 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:54.306240 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-ae6a7a06-d588-4df2-b148-bd5fedcada16 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '20db61fcda1849daafa0b83aaa531324'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:54.307695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-ae6a7a06-d588-4df2-b148-bd5fedcada16 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '20db61fcda1849daafa0b83aaa531324'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:54.309526 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-ae6a7a06-d588-4df2-b148-bd5fedcada16 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '20db61fcda1849daafa0b83aaa531324'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:54.326081 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-ae6a7a06-d588-4df2-b148-bd5fedcada16 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '20db61fcda1849daafa0b83aaa531324'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:54.346570 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-1dd105fe-18dd-4252-b8ec-b57a1a8cd0e0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eacb5c92a3574921b503d6ead653a66d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:54.353590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-ae6a7a06-d588-4df2-b148-bd5fedcada16 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '20db61fcda1849daafa0b83aaa531324'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:54.363609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-1dd105fe-18dd-4252-b8ec-b57a1a8cd0e0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eacb5c92a3574921b503d6ead653a66d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:54.375300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-1dd105fe-18dd-4252-b8ec-b57a1a8cd0e0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eacb5c92a3574921b503d6ead653a66d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:54.375496 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 947/1894] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:30:54 2022] DELETE /identity/v3/users/20db61fcda1849daafa0b83aaa531324 => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:54.407213 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-1dd105fe-18dd-4252-b8ec-b57a1a8cd0e0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eacb5c92a3574921b503d6ead653a66d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:54.408501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-1dd105fe-18dd-4252-b8ec-b57a1a8cd0e0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eacb5c92a3574921b503d6ead653a66d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:54.423012 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 948/1895] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:30:54 2022] DELETE /identity/v3/users/eacb5c92a3574921b503d6ead653a66d => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:54.426903 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ad33f6-ea9e-4433-a427-47af06d7d1e7 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:54.427116 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ad33f6-ea9e-4433-a427-47af06d7d1e7 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:54.427400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ad33f6-ea9e-4433-a427-47af06d7d1e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:54.449244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-18ad33f6-ea9e-4433-a427-47af06d7d1e7 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:54.451418 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-18ad33f6-ea9e-4433-a427-47af06d7d1e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:54.526966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 948/1896] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:54.572765 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd105fe-18dd-4252-b8ec-b57a1a8cd0e0 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:54.575336 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1dd105fe-18dd-4252-b8ec-b57a1a8cd0e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:54.706239 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd105fe-18dd-4252-b8ec-b57a1a8cd0e0 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:54.724764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7129ce5f-21d4-4043-b5a5-160bf9124101 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:54.724764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7129ce5f-21d4-4043-b5a5-160bf9124101 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:54.724764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7129ce5f-21d4-4043-b5a5-160bf9124101 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:54.725034 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7129ce5f-21d4-4043-b5a5-160bf9124101 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:54.747849 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7129ce5f-21d4-4043-b5a5-160bf9124101 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:54.749641 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7129ce5f-21d4-4043-b5a5-160bf9124101 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:54.756626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:54.766204 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 949/1897] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:30:55.306354 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ad33f6-ea9e-4433-a427-47af06d7d1e7 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:55.307878 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-18ad33f6-ea9e-4433-a427-47af06d7d1e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:55.338112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ad33f6-ea9e-4433-a427-47af06d7d1e7 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:55.362197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64be53d6-1795-4cf4-afc6-155c6c9c2f76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:55.362992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64be53d6-1795-4cf4-afc6-155c6c9c2f76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:55.364822 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64be53d6-1795-4cf4-afc6-155c6c9c2f76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:55.364822 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64be53d6-1795-4cf4-afc6-155c6c9c2f76 admin admin] PATH_INFO: `/v3/projects/2ab957150d7c4a269fbf4924e0606c2b` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:55.365881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64be53d6-1795-4cf4-afc6-155c6c9c2f76 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2ab957150d7c4a269fbf4924e0606c2b)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:55.368215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64be53d6-1795-4cf4-afc6-155c6c9c2f76 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:55.405050 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:55.468353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-64be53d6-1795-4cf4-afc6-155c6c9c2f76 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2ab957150d7c4a269fbf4924e0606c2b', 'request_id': 'req-64be53d6-1795-4cf4-afc6-155c6c9c2f76'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:55.476105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-64be53d6-1795-4cf4-afc6-155c6c9c2f76 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2ab957150d7c4a269fbf4924e0606c2b', 'request_id': 'req-64be53d6-1795-4cf4-afc6-155c6c9c2f76'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:55.484200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-64be53d6-1795-4cf4-afc6-155c6c9c2f76 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2ab957150d7c4a269fbf4924e0606c2b', 'request_id': 'req-64be53d6-1795-4cf4-afc6-155c6c9c2f76'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:55.510290 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-64be53d6-1795-4cf4-afc6-155c6c9c2f76 admin admin] The token cache is being invalidate because project 2ab957150d7c4a269fbf4924e0606c2b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:30:55.511112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-64be53d6-1795-4cf4-afc6-155c6c9c2f76 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:55.512172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-64be53d6-1795-4cf4-afc6-155c6c9c2f76 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:55.516394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 949/1898] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:30:55 2022] DELETE /identity/v3/projects/2ab957150d7c4a269fbf4924e0606c2b => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:55.521284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec59e111-ca48-4eac-91b2-454d21338dc0 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:55.522113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec59e111-ca48-4eac-91b2-454d21338dc0 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:55.522113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec59e111-ca48-4eac-91b2-454d21338dc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:55.548297 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-ec59e111-ca48-4eac-91b2-454d21338dc0 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:55.549820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ec59e111-ca48-4eac-91b2-454d21338dc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:55.644085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 950/1899] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:55.658067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8247d2-a0cb-47d5-96eb-30b822dec680 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:55.658582 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8247d2-a0cb-47d5-96eb-30b822dec680 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:55.659097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8247d2-a0cb-47d5-96eb-30b822dec680 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:55.700973 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65753737-1920-45b4-bc91-dd0a09796685 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:55.701381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65753737-1920-45b4-bc91-dd0a09796685 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:55.702753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65753737-1920-45b4-bc91-dd0a09796685 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:55.712550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-0d8247d2-a0cb-47d5-96eb-30b822dec680 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:55.714307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0d8247d2-a0cb-47d5-96eb-30b822dec680 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:55.725384 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 950/1900] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:30:55.731785 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8247d2-a0cb-47d5-96eb-30b822dec680 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:55.742533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8247d2-a0cb-47d5-96eb-30b822dec680 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:55.746853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-65753737-1920-45b4-bc91-dd0a09796685 None None] MFA Rules not processed for user `b3bd51e1acb548ea957ffb57cb09f3f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:55.749018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-65753737-1920-45b4-bc91-dd0a09796685 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:55.752161 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abceebc-5732-44e4-b113-fe5ee11bfed2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:55.752831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abceebc-5732-44e4-b113-fe5ee11bfed2 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:55.753112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abceebc-5732-44e4-b113-fe5ee11bfed2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:55.753388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abceebc-5732-44e4-b113-fe5ee11bfed2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:55.754049 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abceebc-5732-44e4-b113-fe5ee11bfed2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:30:55.754353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abceebc-5732-44e4-b113-fe5ee11bfed2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:55.756577 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abceebc-5732-44e4-b113-fe5ee11bfed2 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:55.773233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 951/1901] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:30:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:30:55.781620 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abceebc-5732-44e4-b113-fe5ee11bfed2 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:55.811562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abceebc-5732-44e4-b113-fe5ee11bfed2 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:55.818907 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fccecb8f-ecb3-444c-bd3c-8fa0c7e944c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:55.819631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccecb8f-ecb3-444c-bd3c-8fa0c7e944c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:55.819991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccecb8f-ecb3-444c-bd3c-8fa0c7e944c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:55.820399 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccecb8f-ecb3-444c-bd3c-8fa0c7e944c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:55.821624 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fccecb8f-ecb3-444c-bd3c-8fa0c7e944c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:55.824548 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fccecb8f-ecb3-444c-bd3c-8fa0c7e944c6 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:55.862789 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 952/1902] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:30:55 2022] POST /identity/v3/projects => generated 407 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:55.863746 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fccecb8f-ecb3-444c-bd3c-8fa0c7e944c6 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:55.871381 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-fccecb8f-ecb3-444c-bd3c-8fa0c7e944c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:55.917790 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 951/1903] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:55 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:55.917790 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65753737-1920-45b4-bc91-dd0a09796685 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:55.951265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65753737-1920-45b4-bc91-dd0a09796685 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:56.014812 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fccecb8f-ecb3-444c-bd3c-8fa0c7e944c6 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:56.023455 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:56.024843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:56.025377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:56.025944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42 admin admin] PATH_INFO: `/v3/projects/5006bacf46684aa9a7b1b5f3d29ee78a` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:56.028643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5006bacf46684aa9a7b1b5f3d29ee78a)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:56.031914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:56.113737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ab7cef-7056-43cc-b9d8-0824c5ef807e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:56.116395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ab7cef-7056-43cc-b9d8-0824c5ef807e admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:56.117122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ab7cef-7056-43cc-b9d8-0824c5ef807e admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:56.117659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ab7cef-7056-43cc-b9d8-0824c5ef807e admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:56.119139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ab7cef-7056-43cc-b9d8-0824c5ef807e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:56.123458 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ab7cef-7056-43cc-b9d8-0824c5ef807e admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:56.140128 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5006bacf46684aa9a7b1b5f3d29ee78a', 'request_id': 'req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:56.150173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5006bacf46684aa9a7b1b5f3d29ee78a', 'request_id': 'req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:56.162409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:56.167691 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5006bacf46684aa9a7b1b5f3d29ee78a', 'request_id': 'req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:56.185169 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 952/1904] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:30:55 2022] POST /identity/v3/users => generated 304 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:56.188515 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ab7cef-7056-43cc-b9d8-0824c5ef807e admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:56.191726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-91ab7cef-7056-43cc-b9d8-0824c5ef807e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:56.198566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42 admin admin] The token cache is being invalidate because project 5006bacf46684aa9a7b1b5f3d29ee78a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:30:56.199165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:56.200363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:30:56.203182 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 953/1905] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:30:55 2022] DELETE /identity/v3/projects/5006bacf46684aa9a7b1b5f3d29ee78a => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:56.204714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:56.207026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:56.242385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:56.327513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ab7cef-7056-43cc-b9d8-0824c5ef807e admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:56.330261 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfa41bc-fcb5-4b06-8ce2-43c4f4bbfc42 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:56.338901 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7192b09c-4827-4b67-b134-a143d054494b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:56.340121 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7192b09c-4827-4b67-b134-a143d054494b admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:56.340438 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7192b09c-4827-4b67-b134-a143d054494b admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:56.340721 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7192b09c-4827-4b67-b134-a143d054494b admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:56.341379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7192b09c-4827-4b67-b134-a143d054494b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:56.343750 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7192b09c-4827-4b67-b134-a143d054494b admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:56.364346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 954/1906] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:56 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:56.369733 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346ec231-14ee-4530-a400-9a6f74b77c94 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:56.370021 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346ec231-14ee-4530-a400-9a6f74b77c94 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:56.370301 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346ec231-14ee-4530-a400-9a6f74b77c94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:56.397313 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-346ec231-14ee-4530-a400-9a6f74b77c94 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:56.398707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-346ec231-14ee-4530-a400-9a6f74b77c94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:56.406260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7bf622-0480-4b96-a372-288d717891a7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:56.407172 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7bf622-0480-4b96-a372-288d717891a7 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:56.407907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7bf622-0480-4b96-a372-288d717891a7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:56.408393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7bf622-0480-4b96-a372-288d717891a7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:56.416720 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 955/1907] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:56.416720 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346ec231-14ee-4530-a400-9a6f74b77c94 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:56.418827 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-aa7bf622-0480-4b96-a372-288d717891a7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:56.425676 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346ec231-14ee-4530-a400-9a6f74b77c94 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:56.432866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fab7c0-8196-47f1-900a-c9aadc6d33fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:56.433750 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fab7c0-8196-47f1-900a-c9aadc6d33fc admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:56.434183 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fab7c0-8196-47f1-900a-c9aadc6d33fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:56.434524 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fab7c0-8196-47f1-900a-c9aadc6d33fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:56.435341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fab7c0-8196-47f1-900a-c9aadc6d33fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:56.437054 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fab7c0-8196-47f1-900a-c9aadc6d33fc admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:56.464172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 956/1908] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:56 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:56.464172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d382777e-74c8-4e13-a3fe-d72353575307 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:56.464172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d382777e-74c8-4e13-a3fe-d72353575307 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:56.464172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d382777e-74c8-4e13-a3fe-d72353575307 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:56.486875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-d382777e-74c8-4e13-a3fe-d72353575307 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:56.488384 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d382777e-74c8-4e13-a3fe-d72353575307 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:56.497755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 957/1909] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:56.499751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d382777e-74c8-4e13-a3fe-d72353575307 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:56.511427 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d382777e-74c8-4e13-a3fe-d72353575307 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:56.517190 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb33c7b4-0469-41c1-8b18-78b56893ad51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:56.518056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb33c7b4-0469-41c1-8b18-78b56893ad51 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:56.518338 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb33c7b4-0469-41c1-8b18-78b56893ad51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:56.518708 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb33c7b4-0469-41c1-8b18-78b56893ad51 admin admin] PATH_INFO: `/v3/projects/89b78c48409445c78c463e61c7fa08f7/users/396a0be62a0d4af2a87814a1be3f951b/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:56.528219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7bf622-0480-4b96-a372-288d717891a7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:56.530638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7bf622-0480-4b96-a372-288d717891a7 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:56.541869 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb33c7b4-0469-41c1-8b18-78b56893ad51 admin admin] RBAC: Authorizing `identity:create_grant(project_id=89b78c48409445c78c463e61c7fa08f7, user_id=396a0be62a0d4af2a87814a1be3f951b, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:56.544005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 953/1910] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:56 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:56.545504 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7bf622-0480-4b96-a372-288d717891a7 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:56.545698 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb33c7b4-0469-41c1-8b18-78b56893ad51 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:56.565736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7bf622-0480-4b96-a372-288d717891a7 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:56.569347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 958/1911] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:56 2022] PUT /identity/v3/projects/89b78c48409445c78c463e61c7fa08f7/users/396a0be62a0d4af2a87814a1be3f951b/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:56.580252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb992480-7397-4a41-98d9-86f7ce56a91a None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:56.580575 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb992480-7397-4a41-98d9-86f7ce56a91a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:56.580851 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb992480-7397-4a41-98d9-86f7ce56a91a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:56.636803 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-eb992480-7397-4a41-98d9-86f7ce56a91a None None] MFA Rules not processed for user `396a0be62a0d4af2a87814a1be3f951b`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:56.639911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-eb992480-7397-4a41-98d9-86f7ce56a91a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:56.705523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a72274-b7c3-46ad-95f4-28b257add28d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:56.706280 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a72274-b7c3-46ad-95f4-28b257add28d admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:56.706556 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a72274-b7c3-46ad-95f4-28b257add28d admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:56.706827 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a72274-b7c3-46ad-95f4-28b257add28d admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:56.709095 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a72274-b7c3-46ad-95f4-28b257add28d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:30:56.709400 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a72274-b7c3-46ad-95f4-28b257add28d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:56.712148 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a72274-b7c3-46ad-95f4-28b257add28d admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:56.733283 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 954/1912] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:30:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:30:56.736085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a72274-b7c3-46ad-95f4-28b257add28d admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:56.749691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a72274-b7c3-46ad-95f4-28b257add28d admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:56.758020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19af593-cbd1-409f-bf3d-7ad6bec8775c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:56.758647 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19af593-cbd1-409f-bf3d-7ad6bec8775c admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:56.758924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19af593-cbd1-409f-bf3d-7ad6bec8775c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:56.759196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19af593-cbd1-409f-bf3d-7ad6bec8775c admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:56.759844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19af593-cbd1-409f-bf3d-7ad6bec8775c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:56.762040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19af593-cbd1-409f-bf3d-7ad6bec8775c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:56.820502 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 959/1913] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:56 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:30:56.820703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 955/1914] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:30:56 2022] POST /identity/v3/projects => generated 397 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:56.822326 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb992480-7397-4a41-98d9-86f7ce56a91a None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:56.825716 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19af593-cbd1-409f-bf3d-7ad6bec8775c admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:56.835090 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:56.868253 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb992480-7397-4a41-98d9-86f7ce56a91a None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:56.922218 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19af593-cbd1-409f-bf3d-7ad6bec8775c admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:56.929458 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577afe12-36ca-46f5-83ca-8a910ffcae04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:56.930608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577afe12-36ca-46f5-83ca-8a910ffcae04 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:56.931084 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577afe12-36ca-46f5-83ca-8a910ffcae04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:56.931537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577afe12-36ca-46f5-83ca-8a910ffcae04 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:56.932493 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577afe12-36ca-46f5-83ca-8a910ffcae04 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:56.935083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577afe12-36ca-46f5-83ca-8a910ffcae04 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:56.956033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:56.980065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 956/1915] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:30:56 2022] POST /identity/v3/projects => generated 407 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:56.981189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577afe12-36ca-46f5-83ca-8a910ffcae04 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:57.013503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f77860-2b93-42e4-947a-ece5038beb7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:57.014240 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f77860-2b93-42e4-947a-ece5038beb7b admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:57.014598 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f77860-2b93-42e4-947a-ece5038beb7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:57.015042 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f77860-2b93-42e4-947a-ece5038beb7b admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:57.016093 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f77860-2b93-42e4-947a-ece5038beb7b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:57.018773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f77860-2b93-42e4-947a-ece5038beb7b admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:57.079352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 960/1916] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:30:56 2022] POST /identity/v3/users => generated 299 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:57.084151 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f77860-2b93-42e4-947a-ece5038beb7b admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:57.096444 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f77860-2b93-42e4-947a-ece5038beb7b admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:57.104865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d79ea9-09f5-4e58-94eb-ae81291a6e5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:57.105587 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d79ea9-09f5-4e58-94eb-ae81291a6e5d admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:57.106950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d79ea9-09f5-4e58-94eb-ae81291a6e5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:57.107304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d79ea9-09f5-4e58-94eb-ae81291a6e5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:57.108408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d79ea9-09f5-4e58-94eb-ae81291a6e5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:57.109621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577afe12-36ca-46f5-83ca-8a910ffcae04 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:57.110415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d79ea9-09f5-4e58-94eb-ae81291a6e5d admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:57.129188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 961/1917] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:57 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:57.131072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d79ea9-09f5-4e58-94eb-ae81291a6e5d admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:57.131212 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2189d52a-3a13-4c70-a995-ab4c45c725fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:57.131777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2189d52a-3a13-4c70-a995-ab4c45c725fb admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:57.132056 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2189d52a-3a13-4c70-a995-ab4c45c725fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:57.132419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2189d52a-3a13-4c70-a995-ab4c45c725fb admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:57.133063 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2189d52a-3a13-4c70-a995-ab4c45c725fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:57.135130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2189d52a-3a13-4c70-a995-ab4c45c725fb admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:57.151534 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d79ea9-09f5-4e58-94eb-ae81291a6e5d admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:57.179171 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ebd78f-248b-4752-bb15-e52392ff0f13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:57.181038 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ebd78f-248b-4752-bb15-e52392ff0f13 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:57.181615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ebd78f-248b-4752-bb15-e52392ff0f13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:57.181873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ebd78f-248b-4752-bb15-e52392ff0f13 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:57.182544 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ebd78f-248b-4752-bb15-e52392ff0f13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:57.184203 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ebd78f-248b-4752-bb15-e52392ff0f13 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:57.196199 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 957/1918] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:30:56 2022] POST /identity/v3/users => generated 310 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:30:57.200033 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2189d52a-3a13-4c70-a995-ab4c45c725fb admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:57.215197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 962/1919] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:57 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:30:57.216092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ebd78f-248b-4752-bb15-e52392ff0f13 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:57.219041 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2189d52a-3a13-4c70-a995-ab4c45c725fb admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:57.236627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ebd78f-248b-4752-bb15-e52392ff0f13 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:57.236627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614a5fd9-8279-48dd-a583-4f7854c4b75c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:57.236627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614a5fd9-8279-48dd-a583-4f7854c4b75c admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:57.236627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614a5fd9-8279-48dd-a583-4f7854c4b75c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:57.238810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614a5fd9-8279-48dd-a583-4f7854c4b75c admin admin] PATH_INFO: `/v3/projects/16e1923f9d6545d3a491436ee1669dae/users/e621f047a40e46b996244ab6fb547900/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:57.240506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35802739-89b7-4a76-aecd-269e8109c835 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:57.241637 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35802739-89b7-4a76-aecd-269e8109c835 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:57.241637 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35802739-89b7-4a76-aecd-269e8109c835 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:57.241836 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35802739-89b7-4a76-aecd-269e8109c835 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:57.242617 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35802739-89b7-4a76-aecd-269e8109c835 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:57.244429 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35802739-89b7-4a76-aecd-269e8109c835 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:57.263714 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 958/1920] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:57 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:30:57.266384 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614a5fd9-8279-48dd-a583-4f7854c4b75c admin admin] RBAC: Authorizing `identity:create_grant(project_id=16e1923f9d6545d3a491436ee1669dae, user_id=e621f047a40e46b996244ab6fb547900, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:57.267535 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35802739-89b7-4a76-aecd-269e8109c835 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:57.272605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614a5fd9-8279-48dd-a583-4f7854c4b75c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:57.280129 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35802739-89b7-4a76-aecd-269e8109c835 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:57.289669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e38579-9837-4768-8ee8-0db8b6898a62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:57.290353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e38579-9837-4768-8ee8-0db8b6898a62 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:57.290639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e38579-9837-4768-8ee8-0db8b6898a62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:57.290915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e38579-9837-4768-8ee8-0db8b6898a62 admin admin] PATH_INFO: `/v3/projects/f2e41d700cc44bf2a5020fe151a7fd8f/users/eed9077efcc14ef696dc801f14cc0f15/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:57.297779 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 963/1921] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:57 2022] PUT /identity/v3/projects/16e1923f9d6545d3a491436ee1669dae/users/e621f047a40e46b996244ab6fb547900/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:30:57.309464 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7912aa2d-f6fa-4a71-8bb1-9258c072fe83 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:57.310026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7912aa2d-f6fa-4a71-8bb1-9258c072fe83 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:57.311261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e38579-9837-4768-8ee8-0db8b6898a62 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2e41d700cc44bf2a5020fe151a7fd8f, user_id=eed9077efcc14ef696dc801f14cc0f15, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:57.311914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7912aa2d-f6fa-4a71-8bb1-9258c072fe83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:57.314301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e38579-9837-4768-8ee8-0db8b6898a62 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:57.335445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 959/1922] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:57 2022] PUT /identity/v3/projects/f2e41d700cc44bf2a5020fe151a7fd8f/users/eed9077efcc14ef696dc801f14cc0f15/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:30:57.339437 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e38579-9837-4768-8ee8-0db8b6898a62 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:57.351702 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-7912aa2d-f6fa-4a71-8bb1-9258c072fe83 None None] MFA Rules not processed for user `e621f047a40e46b996244ab6fb547900`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:57.353039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7912aa2d-f6fa-4a71-8bb1-9258c072fe83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:57.393988 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:57.417745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e38579-9837-4768-8ee8-0db8b6898a62 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:57.422581 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664b6363-7889-44b5-aa2b-66c3a42e8e8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:57.423194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664b6363-7889-44b5-aa2b-66c3a42e8e8b admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:57.423470 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664b6363-7889-44b5-aa2b-66c3a42e8e8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:57.423740 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664b6363-7889-44b5-aa2b-66c3a42e8e8b admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:57.424440 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664b6363-7889-44b5-aa2b-66c3a42e8e8b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:57.426158 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664b6363-7889-44b5-aa2b-66c3a42e8e8b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:57.441176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 960/1923] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:30:57 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:30:57.441896 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 964/1924] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:57 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:57.446229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7912aa2d-f6fa-4a71-8bb1-9258c072fe83 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:57.456571 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664b6363-7889-44b5-aa2b-66c3a42e8e8b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:57.459724 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-664b6363-7889-44b5-aa2b-66c3a42e8e8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:57.467982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7912aa2d-f6fa-4a71-8bb1-9258c072fe83 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:57.473257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa75ed7-b5d1-4832-aadf-dce4a6303cc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:57.474503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa75ed7-b5d1-4832-aadf-dce4a6303cc7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:57.474806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa75ed7-b5d1-4832-aadf-dce4a6303cc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:57.475110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa75ed7-b5d1-4832-aadf-dce4a6303cc7 admin admin] PATH_INFO: `/v3/projects/f2e41d700cc44bf2a5020fe151a7fd8f/users/eed9077efcc14ef696dc801f14cc0f15/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:57.477224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa75ed7-b5d1-4832-aadf-dce4a6303cc7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2e41d700cc44bf2a5020fe151a7fd8f, user_id=eed9077efcc14ef696dc801f14cc0f15, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:57.481301 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa75ed7-b5d1-4832-aadf-dce4a6303cc7 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:57.497535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 965/1925] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:30:57 2022] PUT /identity/v3/projects/f2e41d700cc44bf2a5020fe151a7fd8f/users/eed9077efcc14ef696dc801f14cc0f15/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:30:57.509750 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb7d8e7-de97-4884-8fad-39a2783f4866 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:57.510103 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb7d8e7-de97-4884-8fad-39a2783f4866 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:57.510364 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb7d8e7-de97-4884-8fad-39a2783f4866 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:57.544774 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664b6363-7889-44b5-aa2b-66c3a42e8e8b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:57.545822 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-edb7d8e7-de97-4884-8fad-39a2783f4866 None None] MFA Rules not processed for user `eed9077efcc14ef696dc801f14cc0f15`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:57.547118 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-edb7d8e7-de97-4884-8fad-39a2783f4866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:57.591849 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:57.610180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35095994-f139-4a5a-8145-388de96ef6b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:57.610798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35095994-f139-4a5a-8145-388de96ef6b0 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:57.611072 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35095994-f139-4a5a-8145-388de96ef6b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:57.611337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35095994-f139-4a5a-8145-388de96ef6b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:57.651237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 966/1926] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:57 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:30:57.679314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb7d8e7-de97-4884-8fad-39a2783f4866 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:57.681135 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-edb7d8e7-de97-4884-8fad-39a2783f4866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:57.694834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb7d8e7-de97-4884-8fad-39a2783f4866 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:57.700594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35095994-f139-4a5a-8145-388de96ef6b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:57.702144 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35095994-f139-4a5a-8145-388de96ef6b0 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:57.709020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbb57bd-977a-48b1-8720-b83cfe5a2242 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:57.709952 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbb57bd-977a-48b1-8720-b83cfe5a2242 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:57.711484 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbb57bd-977a-48b1-8720-b83cfe5a2242 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:57.711979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbb57bd-977a-48b1-8720-b83cfe5a2242 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:57.717541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 961/1927] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:57 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 262 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:57.728921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbb57bd-977a-48b1-8720-b83cfe5a2242 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:57.731326 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbb57bd-977a-48b1-8720-b83cfe5a2242 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:57.743293 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 967/1928] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:57 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:30:57.964429 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35095994-f139-4a5a-8145-388de96ef6b0 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:30:57.973107 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35095994-f139-4a5a-8145-388de96ef6b0 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:30:57.978886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc2c8df-6834-4bf2-aa80-838f27e114e2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:30:57.979563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc2c8df-6834-4bf2-aa80-838f27e114e2 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:57.979882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc2c8df-6834-4bf2-aa80-838f27e114e2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:57.980123 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc2c8df-6834-4bf2-aa80-838f27e114e2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:58.043724 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:30:58.063281 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc2c8df-6834-4bf2-aa80-838f27e114e2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:30:58.064790 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc2c8df-6834-4bf2-aa80-838f27e114e2 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:30:58.076395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 962/1929] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:30:57 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:30:58.318795 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa07f2ee-2872-402c-9d17-d606b3624870 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:30:58.319051 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa07f2ee-2872-402c-9d17-d606b3624870 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:30:58.319348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa07f2ee-2872-402c-9d17-d606b3624870 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:30:58.355735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-aa07f2ee-2872-402c-9d17-d606b3624870 None None] MFA Rules not processed for user `b3bd51e1acb548ea957ffb57cb09f3f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:30:58.357264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-aa07f2ee-2872-402c-9d17-d606b3624870 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:30:58.439397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 968/1930] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:30:58 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:08.079516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=6 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:08.194889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc2c8df-6834-4bf2-aa80-838f27e114e2 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:08.291081 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc2c8df-6834-4bf2-aa80-838f27e114e2 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:08.295669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd83cc95-8f44-497f-b5d3-5ae2ddc7c5fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:08.296296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd83cc95-8f44-497f-b5d3-5ae2ddc7c5fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:08.296595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd83cc95-8f44-497f-b5d3-5ae2ddc7c5fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:08.296860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd83cc95-8f44-497f-b5d3-5ae2ddc7c5fc admin admin] PATH_INFO: `/v3/users/e621f047a40e46b996244ab6fb547900` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:08.297999 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd83cc95-8f44-497f-b5d3-5ae2ddc7c5fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=e621f047a40e46b996244ab6fb547900)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:08.299913 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd83cc95-8f44-497f-b5d3-5ae2ddc7c5fc admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:08.353418 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cd83cc95-8f44-497f-b5d3-5ae2ddc7c5fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e621f047a40e46b996244ab6fb547900'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:08.361858 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cd83cc95-8f44-497f-b5d3-5ae2ddc7c5fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e621f047a40e46b996244ab6fb547900'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:08.370688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cd83cc95-8f44-497f-b5d3-5ae2ddc7c5fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e621f047a40e46b996244ab6fb547900'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:08.384760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cd83cc95-8f44-497f-b5d3-5ae2ddc7c5fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e621f047a40e46b996244ab6fb547900'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:08.385516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cd83cc95-8f44-497f-b5d3-5ae2ddc7c5fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e621f047a40e46b996244ab6fb547900'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:08.392133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 963/1931] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:31:08 2022] DELETE /identity/v3/users/e621f047a40e46b996244ab6fb547900 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:08.397056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553bfdc5-6231-48ee-b507-66dde8ea858c None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:08.397599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553bfdc5-6231-48ee-b507-66dde8ea858c None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:08.397599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553bfdc5-6231-48ee-b507-66dde8ea858c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:08.419501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-553bfdc5-6231-48ee-b507-66dde8ea858c None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:08.420812 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-553bfdc5-6231-48ee-b507-66dde8ea858c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:08.490837 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 969/1932] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:08.501172 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd83cc95-8f44-497f-b5d3-5ae2ddc7c5fc admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:08.502813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-cd83cc95-8f44-497f-b5d3-5ae2ddc7c5fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:08.576056 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd83cc95-8f44-497f-b5d3-5ae2ddc7c5fc admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:08.579861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4818ccc-12a5-4e54-b3b3-daccbae59158 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:08.580509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4818ccc-12a5-4e54-b3b3-daccbae59158 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:08.580791 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4818ccc-12a5-4e54-b3b3-daccbae59158 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:08.581061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4818ccc-12a5-4e54-b3b3-daccbae59158 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:08.592001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4818ccc-12a5-4e54-b3b3-daccbae59158 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:08.593526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4818ccc-12a5-4e54-b3b3-daccbae59158 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:08.605133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 964/1933] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:31:09.104399 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553bfdc5-6231-48ee-b507-66dde8ea858c None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:09.106176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-553bfdc5-6231-48ee-b507-66dde8ea858c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:09.119992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553bfdc5-6231-48ee-b507-66dde8ea858c None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:09.125252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64408467-2cea-4422-86f0-ff68e0d31979 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:09.125898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64408467-2cea-4422-86f0-ff68e0d31979 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:09.126178 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64408467-2cea-4422-86f0-ff68e0d31979 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:09.126451 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64408467-2cea-4422-86f0-ff68e0d31979 admin admin] PATH_INFO: `/v3/projects/16e1923f9d6545d3a491436ee1669dae` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:09.127591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64408467-2cea-4422-86f0-ff68e0d31979 admin admin] RBAC: Authorizing `identity:delete_project(project_id=16e1923f9d6545d3a491436ee1669dae)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:09.129785 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64408467-2cea-4422-86f0-ff68e0d31979 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:09.217503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-64408467-2cea-4422-86f0-ff68e0d31979 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '16e1923f9d6545d3a491436ee1669dae', 'request_id': 'req-64408467-2cea-4422-86f0-ff68e0d31979'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:09.225574 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-64408467-2cea-4422-86f0-ff68e0d31979 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '16e1923f9d6545d3a491436ee1669dae', 'request_id': 'req-64408467-2cea-4422-86f0-ff68e0d31979'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:09.232221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:09.235878 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-64408467-2cea-4422-86f0-ff68e0d31979 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '16e1923f9d6545d3a491436ee1669dae', 'request_id': 'req-64408467-2cea-4422-86f0-ff68e0d31979'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:09.275422 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-64408467-2cea-4422-86f0-ff68e0d31979 admin admin] The token cache is being invalidate because project 16e1923f9d6545d3a491436ee1669dae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:31:09.275638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-64408467-2cea-4422-86f0-ff68e0d31979 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:09.276391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-64408467-2cea-4422-86f0-ff68e0d31979 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:09.283529 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 970/1934] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:31:09 2022] DELETE /identity/v3/projects/16e1923f9d6545d3a491436ee1669dae => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:09.286548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb85601-f916-4fb2-851d-2bbec18a8a90 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:09.287084 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb85601-f916-4fb2-851d-2bbec18a8a90 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:09.287717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb85601-f916-4fb2-851d-2bbec18a8a90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:09.321004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-cdb85601-f916-4fb2-851d-2bbec18a8a90 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:09.322707 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-cdb85601-f916-4fb2-851d-2bbec18a8a90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:09.347007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:09.425714 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 965/1935] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:09.436081 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96233a0f-d733-439f-83a6-a6a035ccf8fd None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:09.436663 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96233a0f-d733-439f-83a6-a6a035ccf8fd None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:09.436792 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96233a0f-d733-439f-83a6-a6a035ccf8fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:09.471207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-96233a0f-d733-439f-83a6-a6a035ccf8fd None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:09.472440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-96233a0f-d733-439f-83a6-a6a035ccf8fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:09.480896 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 971/1936] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:09.485588 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb85601-f916-4fb2-851d-2bbec18a8a90 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:09.496284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb85601-f916-4fb2-851d-2bbec18a8a90 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:09.501801 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46eb886b-63d4-42da-8ea9-e0007b11b355 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:09.502540 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eb886b-63d4-42da-8ea9-e0007b11b355 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:09.502905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eb886b-63d4-42da-8ea9-e0007b11b355 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:09.503257 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eb886b-63d4-42da-8ea9-e0007b11b355 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:09.503999 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46eb886b-63d4-42da-8ea9-e0007b11b355 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:31:09.504985 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46eb886b-63d4-42da-8ea9-e0007b11b355 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:09.506683 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46eb886b-63d4-42da-8ea9-e0007b11b355 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:09.519586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 966/1937] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:31:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:31:09.524400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96233a0f-d733-439f-83a6-a6a035ccf8fd None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:09.534768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96233a0f-d733-439f-83a6-a6a035ccf8fd None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:09.539980 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf85ee10-1f11-4c48-b349-f6a392f5038a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:09.540650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf85ee10-1f11-4c48-b349-f6a392f5038a admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:09.540933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf85ee10-1f11-4c48-b349-f6a392f5038a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:09.541201 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf85ee10-1f11-4c48-b349-f6a392f5038a admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:09.542052 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf85ee10-1f11-4c48-b349-f6a392f5038a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:09.544399 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf85ee10-1f11-4c48-b349-f6a392f5038a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:09.576183 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 972/1938] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:31:09 2022] POST /identity/v3/projects => generated 379 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:31:09.580615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46eb886b-63d4-42da-8ea9-e0007b11b355 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:09.593138 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46eb886b-63d4-42da-8ea9-e0007b11b355 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:09.667816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d070af7-3f5c-45c0-9fd3-97f951f02f73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:09.668478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d070af7-3f5c-45c0-9fd3-97f951f02f73 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:09.668791 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d070af7-3f5c-45c0-9fd3-97f951f02f73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:09.669066 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d070af7-3f5c-45c0-9fd3-97f951f02f73 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:09.669706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d070af7-3f5c-45c0-9fd3-97f951f02f73 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:09.671650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d070af7-3f5c-45c0-9fd3-97f951f02f73 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:09.708275 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 967/1939] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:31:09 2022] POST /identity/v3/users => generated 290 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:09.710201 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf85ee10-1f11-4c48-b349-f6a392f5038a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:09.725791 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf85ee10-1f11-4c48-b349-f6a392f5038a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:09.732951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef5142d-d52e-45d2-b072-cea340c3fc9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:09.733577 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef5142d-d52e-45d2-b072-cea340c3fc9f admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:09.734415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef5142d-d52e-45d2-b072-cea340c3fc9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:09.734415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef5142d-d52e-45d2-b072-cea340c3fc9f admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:09.734934 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef5142d-d52e-45d2-b072-cea340c3fc9f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:09.736559 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef5142d-d52e-45d2-b072-cea340c3fc9f admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:09.751987 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 973/1940] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:09 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:09.753848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d070af7-3f5c-45c0-9fd3-97f951f02f73 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:09.777026 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d070af7-3f5c-45c0-9fd3-97f951f02f73 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:09.781752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9431c85-32cc-46f3-a2a4-7b890a99d63d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:09.782425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9431c85-32cc-46f3-a2a4-7b890a99d63d admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:09.782967 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9431c85-32cc-46f3-a2a4-7b890a99d63d admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:09.783236 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9431c85-32cc-46f3-a2a4-7b890a99d63d admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:09.783894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9431c85-32cc-46f3-a2a4-7b890a99d63d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:09.785583 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9431c85-32cc-46f3-a2a4-7b890a99d63d admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:09.797397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 968/1941] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:09 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:09.801729 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef5142d-d52e-45d2-b072-cea340c3fc9f admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:09.813010 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef5142d-d52e-45d2-b072-cea340c3fc9f admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:09.818858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd63297c-2190-4ca3-b01c-6becaa47d408 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:09.819674 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd63297c-2190-4ca3-b01c-6becaa47d408 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:09.820061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd63297c-2190-4ca3-b01c-6becaa47d408 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:09.820458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd63297c-2190-4ca3-b01c-6becaa47d408 admin admin] PATH_INFO: `/v3/projects/408b16a582484c619ff17416990d5feb/users/c7028d2ccede4b44b567387bb83c4579/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:09.837563 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd63297c-2190-4ca3-b01c-6becaa47d408 admin admin] RBAC: Authorizing `identity:create_grant(project_id=408b16a582484c619ff17416990d5feb, user_id=c7028d2ccede4b44b567387bb83c4579, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:09.840411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd63297c-2190-4ca3-b01c-6becaa47d408 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:09.844350 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9431c85-32cc-46f3-a2a4-7b890a99d63d admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:09.846055 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b9431c85-32cc-46f3-a2a4-7b890a99d63d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:09.856248 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 974/1942] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:09 2022] PUT /identity/v3/projects/408b16a582484c619ff17416990d5feb/users/c7028d2ccede4b44b567387bb83c4579/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:31:09.859666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9431c85-32cc-46f3-a2a4-7b890a99d63d admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:09.861431 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba288c34-7eb4-48fc-a096-96b090020ccc None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:09.861771 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba288c34-7eb4-48fc-a096-96b090020ccc None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:09.862085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba288c34-7eb4-48fc-a096-96b090020ccc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:09.882968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-ba288c34-7eb4-48fc-a096-96b090020ccc None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:09.884303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ba288c34-7eb4-48fc-a096-96b090020ccc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:09.921705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8316b85e-61f6-4011-9bee-257e03ea0aed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:09.922341 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8316b85e-61f6-4011-9bee-257e03ea0aed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:09.922623 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8316b85e-61f6-4011-9bee-257e03ea0aed admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:09.922894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8316b85e-61f6-4011-9bee-257e03ea0aed admin admin] PATH_INFO: `/v3/users/396a0be62a0d4af2a87814a1be3f951b` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:09.924112 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8316b85e-61f6-4011-9bee-257e03ea0aed admin admin] RBAC: Authorizing `identity:delete_user(user_id=396a0be62a0d4af2a87814a1be3f951b)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:09.926099 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8316b85e-61f6-4011-9bee-257e03ea0aed admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:09.950302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 975/1943] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:09.961314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba288c34-7eb4-48fc-a096-96b090020ccc None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:09.962802 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ba288c34-7eb4-48fc-a096-96b090020ccc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:09.964271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:09.983708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8316b85e-61f6-4011-9bee-257e03ea0aed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '396a0be62a0d4af2a87814a1be3f951b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:09.993220 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8316b85e-61f6-4011-9bee-257e03ea0aed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '396a0be62a0d4af2a87814a1be3f951b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:10.001208 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8316b85e-61f6-4011-9bee-257e03ea0aed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '396a0be62a0d4af2a87814a1be3f951b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:10.016574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8316b85e-61f6-4011-9bee-257e03ea0aed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '396a0be62a0d4af2a87814a1be3f951b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:10.017284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8316b85e-61f6-4011-9bee-257e03ea0aed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '396a0be62a0d4af2a87814a1be3f951b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:10.024501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 969/1944] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:31:09 2022] DELETE /identity/v3/users/396a0be62a0d4af2a87814a1be3f951b => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:31:10.025801 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba288c34-7eb4-48fc-a096-96b090020ccc None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:10.029094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5723efd-2249-4b14-b18f-258b1eb230dd None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:10.029382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5723efd-2249-4b14-b18f-258b1eb230dd None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:10.029670 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5723efd-2249-4b14-b18f-258b1eb230dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:10.052446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-a5723efd-2249-4b14-b18f-258b1eb230dd None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:10.053881 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a5723efd-2249-4b14-b18f-258b1eb230dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:10.071796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:DELETE=2,keystone:SELECT=95,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:10.087058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969c6547-bd17-47a5-a185-2570459075ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:10.087695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969c6547-bd17-47a5-a185-2570459075ad service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:10.087978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969c6547-bd17-47a5-a185-2570459075ad service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:10.088255 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969c6547-bd17-47a5-a185-2570459075ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:10.090292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-969c6547-bd17-47a5-a185-2570459075ad service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:10.123118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 970/1945] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:10.133104 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5723efd-2249-4b14-b18f-258b1eb230dd None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:10.134644 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a5723efd-2249-4b14-b18f-258b1eb230dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:10.147612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5723efd-2249-4b14-b18f-258b1eb230dd None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:10.151466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d362e6-fd3a-4c0b-943d-552f6aac7e8d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:10.152073 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d362e6-fd3a-4c0b-943d-552f6aac7e8d service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:10.152373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d362e6-fd3a-4c0b-943d-552f6aac7e8d service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:10.152644 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d362e6-fd3a-4c0b-943d-552f6aac7e8d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:10.164158 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d362e6-fd3a-4c0b-943d-552f6aac7e8d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:10.165643 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d362e6-fd3a-4c0b-943d-552f6aac7e8d service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:10.174016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969c6547-bd17-47a5-a185-2570459075ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:10.175493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969c6547-bd17-47a5-a185-2570459075ad service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:10.176903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 971/1946] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:31:10.186406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 976/1947] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:31:10.838781 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d362e6-fd3a-4c0b-943d-552f6aac7e8d service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:10.840703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-33d362e6-fd3a-4c0b-943d-552f6aac7e8d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:10.854710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d362e6-fd3a-4c0b-943d-552f6aac7e8d service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:10.860058 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:10.860952 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:10.861228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:10.861500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c admin admin] PATH_INFO: `/v3/projects/89b78c48409445c78c463e61c7fa08f7` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:10.862644 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c admin admin] RBAC: Authorizing `identity:delete_project(project_id=89b78c48409445c78c463e61c7fa08f7)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:10.865480 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:10.920911 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '89b78c48409445c78c463e61c7fa08f7', 'request_id': 'req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:10.931122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '89b78c48409445c78c463e61c7fa08f7', 'request_id': 'req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:10.943059 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '89b78c48409445c78c463e61c7fa08f7', 'request_id': 'req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:10.959086 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c admin admin] The token cache is being invalidate because project 89b78c48409445c78c463e61c7fa08f7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:31:10.959486 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:10.960179 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:10.961748 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 972/1948] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:31:10 2022] DELETE /identity/v3/projects/89b78c48409445c78c463e61c7fa08f7 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:10.966058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969c6547-bd17-47a5-a185-2570459075ad service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:10.978034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969c6547-bd17-47a5-a185-2570459075ad service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:11.042944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3621fe1c-1f7d-4ae7-8d8f-060e8d382378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:11.043605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3621fe1c-1f7d-4ae7-8d8f-060e8d382378 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:11.043890 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3621fe1c-1f7d-4ae7-8d8f-060e8d382378 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:11.044165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3621fe1c-1f7d-4ae7-8d8f-060e8d382378 admin admin] PATH_INFO: `/v3/users/eed9077efcc14ef696dc801f14cc0f15` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:11.045369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3621fe1c-1f7d-4ae7-8d8f-060e8d382378 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eed9077efcc14ef696dc801f14cc0f15)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:11.047384 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3621fe1c-1f7d-4ae7-8d8f-060e8d382378 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:11.107197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3621fe1c-1f7d-4ae7-8d8f-060e8d382378 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eed9077efcc14ef696dc801f14cc0f15'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:11.108106 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3621fe1c-1f7d-4ae7-8d8f-060e8d382378 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eed9077efcc14ef696dc801f14cc0f15'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:11.108915 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3621fe1c-1f7d-4ae7-8d8f-060e8d382378 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eed9077efcc14ef696dc801f14cc0f15'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:11.118888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3621fe1c-1f7d-4ae7-8d8f-060e8d382378 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eed9077efcc14ef696dc801f14cc0f15'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:11.135891 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3621fe1c-1f7d-4ae7-8d8f-060e8d382378 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eed9077efcc14ef696dc801f14cc0f15'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:11.157369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 977/1949] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:31:10 2022] DELETE /identity/v3/users/eed9077efcc14ef696dc801f14cc0f15 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:11.676723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:11.678252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:11.712497 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c56c84a-4ae5-4fc9-a6d5-deb2d2f8cc5c admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:11.734453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:11.831955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076bb955-65e0-473f-90ea-58103f214e28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:11.833563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076bb955-65e0-473f-90ea-58103f214e28 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:11.833831 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076bb955-65e0-473f-90ea-58103f214e28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:11.834988 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076bb955-65e0-473f-90ea-58103f214e28 admin admin] PATH_INFO: `/v3/projects/f2e41d700cc44bf2a5020fe151a7fd8f` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:11.836630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076bb955-65e0-473f-90ea-58103f214e28 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f2e41d700cc44bf2a5020fe151a7fd8f)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:11.839021 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076bb955-65e0-473f-90ea-58103f214e28 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:11.904442 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-076bb955-65e0-473f-90ea-58103f214e28 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f2e41d700cc44bf2a5020fe151a7fd8f', 'request_id': 'req-076bb955-65e0-473f-90ea-58103f214e28'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:11.912357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-076bb955-65e0-473f-90ea-58103f214e28 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f2e41d700cc44bf2a5020fe151a7fd8f', 'request_id': 'req-076bb955-65e0-473f-90ea-58103f214e28'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:11.919516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-076bb955-65e0-473f-90ea-58103f214e28 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f2e41d700cc44bf2a5020fe151a7fd8f', 'request_id': 'req-076bb955-65e0-473f-90ea-58103f214e28'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:11.939772 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-076bb955-65e0-473f-90ea-58103f214e28 admin admin] The token cache is being invalidate because project f2e41d700cc44bf2a5020fe151a7fd8f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:31:11.940232 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-076bb955-65e0-473f-90ea-58103f214e28 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:11.941171 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-076bb955-65e0-473f-90ea-58103f214e28 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:11.946785 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 973/1950] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:31:11 2022] DELETE /identity/v3/projects/f2e41d700cc44bf2a5020fe151a7fd8f => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:11.948605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f715c0-1785-45be-a1d6-7612970f0164 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:11.948863 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f715c0-1785-45be-a1d6-7612970f0164 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:11.949152 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f715c0-1785-45be-a1d6-7612970f0164 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:11.977172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-20f715c0-1785-45be-a1d6-7612970f0164 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:11.979102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-20f715c0-1785-45be-a1d6-7612970f0164 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:11.990435 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:12.064441 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 978/1951] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:12.073782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd0e890-cde0-4b18-9496-0da2efd3433d None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:12.074072 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd0e890-cde0-4b18-9496-0da2efd3433d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:12.074369 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd0e890-cde0-4b18-9496-0da2efd3433d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:12.096427 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-1fd0e890-cde0-4b18-9496-0da2efd3433d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:12.097647 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1fd0e890-cde0-4b18-9496-0da2efd3433d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:12.106291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 974/1952] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:12.113108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f715c0-1785-45be-a1d6-7612970f0164 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:12.122915 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f715c0-1785-45be-a1d6-7612970f0164 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:12.129238 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbd7361-37f3-43fa-8c7c-0667d0c23a41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:12.129967 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbd7361-37f3-43fa-8c7c-0667d0c23a41 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:12.130310 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbd7361-37f3-43fa-8c7c-0667d0c23a41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:12.130650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbd7361-37f3-43fa-8c7c-0667d0c23a41 admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:12.131599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbd7361-37f3-43fa-8c7c-0667d0c23a41 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:31:12.131719 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbd7361-37f3-43fa-8c7c-0667d0c23a41 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:12.133526 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbd7361-37f3-43fa-8c7c-0667d0c23a41 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:12.151253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd0e890-cde0-4b18-9496-0da2efd3433d None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:12.152454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 979/1953] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:31:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:31:12.162640 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd0e890-cde0-4b18-9496-0da2efd3433d None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:12.168635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a10f31-2a0b-4eac-865a-54e5e05ead0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:12.168926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a10f31-2a0b-4eac-865a-54e5e05ead0b admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:12.169108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a10f31-2a0b-4eac-865a-54e5e05ead0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:12.169474 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a10f31-2a0b-4eac-865a-54e5e05ead0b admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:12.170187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a10f31-2a0b-4eac-865a-54e5e05ead0b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:12.172346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a10f31-2a0b-4eac-865a-54e5e05ead0b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:12.209497 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 975/1954] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:31:12 2022] POST /identity/v3/projects => generated 371 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:31:12.210800 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbd7361-37f3-43fa-8c7c-0667d0c23a41 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:12.251261 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbd7361-37f3-43fa-8c7c-0667d0c23a41 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:12.333118 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152a6b09-ff1d-41d5-b134-792dfe9094e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:12.333845 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152a6b09-ff1d-41d5-b134-792dfe9094e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:12.334493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152a6b09-ff1d-41d5-b134-792dfe9094e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:12.334836 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152a6b09-ff1d-41d5-b134-792dfe9094e2 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:12.336543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152a6b09-ff1d-41d5-b134-792dfe9094e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:12.337678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152a6b09-ff1d-41d5-b134-792dfe9094e2 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:12.372556 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 980/1955] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:31:12 2022] POST /identity/v3/users => generated 286 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:12.376847 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a10f31-2a0b-4eac-865a-54e5e05ead0b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:12.401786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a10f31-2a0b-4eac-865a-54e5e05ead0b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:12.406536 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb7e115-8e9b-4b5e-91c3-48354dfbbb7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:12.407068 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb7e115-8e9b-4b5e-91c3-48354dfbbb7b admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:12.407346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb7e115-8e9b-4b5e-91c3-48354dfbbb7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:12.407882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb7e115-8e9b-4b5e-91c3-48354dfbbb7b admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:12.408582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb7e115-8e9b-4b5e-91c3-48354dfbbb7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:12.410204 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb7e115-8e9b-4b5e-91c3-48354dfbbb7b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:12.421327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 976/1956] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:12 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:31:12.425313 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152a6b09-ff1d-41d5-b134-792dfe9094e2 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:12.436870 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152a6b09-ff1d-41d5-b134-792dfe9094e2 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:12.443011 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd28038-00c9-43de-b600-2ede5c747dc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:12.443628 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd28038-00c9-43de-b600-2ede5c747dc1 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:12.443916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd28038-00c9-43de-b600-2ede5c747dc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:12.444193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd28038-00c9-43de-b600-2ede5c747dc1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:12.444921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd28038-00c9-43de-b600-2ede5c747dc1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:12.446601 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd28038-00c9-43de-b600-2ede5c747dc1 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:12.457593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 981/1957] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:12 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:31:12.461915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb7e115-8e9b-4b5e-91c3-48354dfbbb7b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:12.474314 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb7e115-8e9b-4b5e-91c3-48354dfbbb7b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:12.480746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af095cba-2ca3-4b74-ae56-93e513bdfcbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:12.481657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af095cba-2ca3-4b74-ae56-93e513bdfcbb admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:12.482079 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af095cba-2ca3-4b74-ae56-93e513bdfcbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:12.482514 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af095cba-2ca3-4b74-ae56-93e513bdfcbb admin admin] PATH_INFO: `/v3/projects/29cbd816b9e24c17b3d0412e7cd85328/users/14df6dfb84e743268ff8cd5d7600c759/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:12.504227 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af095cba-2ca3-4b74-ae56-93e513bdfcbb admin admin] RBAC: Authorizing `identity:create_grant(project_id=29cbd816b9e24c17b3d0412e7cd85328, user_id=14df6dfb84e743268ff8cd5d7600c759, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:12.507416 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af095cba-2ca3-4b74-ae56-93e513bdfcbb admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:12.523528 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 977/1958] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:12 2022] PUT /identity/v3/projects/29cbd816b9e24c17b3d0412e7cd85328/users/14df6dfb84e743268ff8cd5d7600c759/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:12.528871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6750810d-ef3c-4621-9861-4031282f107d None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:12.529165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6750810d-ef3c-4621-9861-4031282f107d None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:12.529446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6750810d-ef3c-4621-9861-4031282f107d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:12.551903 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-6750810d-ef3c-4621-9861-4031282f107d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:12.553272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-6750810d-ef3c-4621-9861-4031282f107d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:12.619825 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 982/1959] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:12.631560 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af095cba-2ca3-4b74-ae56-93e513bdfcbb admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:12.633395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-af095cba-2ca3-4b74-ae56-93e513bdfcbb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:12.685422 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:12.698608 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af095cba-2ca3-4b74-ae56-93e513bdfcbb admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:12.702928 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd4d2b1-960a-44cb-bdde-a794e69eb0fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:12.703693 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd4d2b1-960a-44cb-bdde-a794e69eb0fb service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:12.704077 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd4d2b1-960a-44cb-bdde-a794e69eb0fb service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:12.704525 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd4d2b1-960a-44cb-bdde-a794e69eb0fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:12.714734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd4d2b1-960a-44cb-bdde-a794e69eb0fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:12.716138 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd4d2b1-960a-44cb-bdde-a794e69eb0fb service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:12.727618 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 978/1960] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:31:13.759239 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab67c6d3-e834-4e01-9b34-03895c7af1e8 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:13.759869 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab67c6d3-e834-4e01-9b34-03895c7af1e8 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:13.759869 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab67c6d3-e834-4e01-9b34-03895c7af1e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:13.806187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-ab67c6d3-e834-4e01-9b34-03895c7af1e8 None None] MFA Rules not processed for user `14df6dfb84e743268ff8cd5d7600c759`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:13.807462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ab67c6d3-e834-4e01-9b34-03895c7af1e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:13.812062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:13.921353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 983/1961] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:13 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:13.926595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd4d2b1-960a-44cb-bdde-a794e69eb0fb service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:13.951421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd4d2b1-960a-44cb-bdde-a794e69eb0fb service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:13.958543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f94bbc2-e7ca-4c16-88df-44d85219f81a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:13.959460 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f94bbc2-e7ca-4c16-88df-44d85219f81a admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:13.959745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f94bbc2-e7ca-4c16-88df-44d85219f81a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:13.960021 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f94bbc2-e7ca-4c16-88df-44d85219f81a admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:13.960716 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f94bbc2-e7ca-4c16-88df-44d85219f81a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:13.963624 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f94bbc2-e7ca-4c16-88df-44d85219f81a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:14.005078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 979/1962] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:31:13 2022] POST /identity/v3/projects => generated 373 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:14.008749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab67c6d3-e834-4e01-9b34-03895c7af1e8 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:14.022971 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab67c6d3-e834-4e01-9b34-03895c7af1e8 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:14.121339 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfee920-d7d6-4560-8fe0-d8bb72419288 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:14.122012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfee920-d7d6-4560-8fe0-d8bb72419288 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:14.122295 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfee920-d7d6-4560-8fe0-d8bb72419288 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:14.122562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfee920-d7d6-4560-8fe0-d8bb72419288 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:14.123857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfee920-d7d6-4560-8fe0-d8bb72419288 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:14.126805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfee920-d7d6-4560-8fe0-d8bb72419288 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:14.160677 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 984/1963] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:31:14 2022] POST /identity/v3/users => generated 293 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:14.164532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f94bbc2-e7ca-4c16-88df-44d85219f81a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:14.179359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f94bbc2-e7ca-4c16-88df-44d85219f81a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:14.185386 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9635e250-9646-4cac-b71b-7d31f252a5ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:14.187345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9635e250-9646-4cac-b71b-7d31f252a5ce admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:14.187818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9635e250-9646-4cac-b71b-7d31f252a5ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:14.188272 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9635e250-9646-4cac-b71b-7d31f252a5ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:14.190577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9635e250-9646-4cac-b71b-7d31f252a5ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:14.192989 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9635e250-9646-4cac-b71b-7d31f252a5ce admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:14.215883 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 980/1964] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:14 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:14.219623 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfee920-d7d6-4560-8fe0-d8bb72419288 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:14.232508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfee920-d7d6-4560-8fe0-d8bb72419288 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:14.237662 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ecfa19-7218-4177-9a0a-7fa1b650311a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:14.238306 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ecfa19-7218-4177-9a0a-7fa1b650311a admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:14.239187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ecfa19-7218-4177-9a0a-7fa1b650311a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:14.239677 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ecfa19-7218-4177-9a0a-7fa1b650311a admin admin] PATH_INFO: `/v3/projects/69bc01fd66694a188ec00074d35ea687/users/2b48ebda9b7648bebb4fa4712ab3c17c/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:14.261143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ecfa19-7218-4177-9a0a-7fa1b650311a admin admin] RBAC: Authorizing `identity:create_grant(project_id=69bc01fd66694a188ec00074d35ea687, user_id=2b48ebda9b7648bebb4fa4712ab3c17c, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:14.264271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ecfa19-7218-4177-9a0a-7fa1b650311a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:14.324539 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 985/1965] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:14 2022] PUT /identity/v3/projects/69bc01fd66694a188ec00074d35ea687/users/2b48ebda9b7648bebb4fa4712ab3c17c/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:14.329534 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9635e250-9646-4cac-b71b-7d31f252a5ce admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:14.355735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9635e250-9646-4cac-b71b-7d31f252a5ce admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:14.468636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c4cb11-07eb-4b3a-ae5a-20b05fd15ad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:14.469545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c4cb11-07eb-4b3a-ae5a-20b05fd15ad4 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:14.469984 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c4cb11-07eb-4b3a-ae5a-20b05fd15ad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:14.477640 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c4cb11-07eb-4b3a-ae5a-20b05fd15ad4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:14.478313 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c4cb11-07eb-4b3a-ae5a-20b05fd15ad4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:14.480304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c4cb11-07eb-4b3a-ae5a-20b05fd15ad4 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:14.506689 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 981/1966] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:14 2022] GET /identity/v3/roles => generated 1420 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:14.507929 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ecfa19-7218-4177-9a0a-7fa1b650311a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:14.521714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ecfa19-7218-4177-9a0a-7fa1b650311a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:14.526531 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fea539a-ac7f-484d-adc3-c4e173460544 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:14.527198 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fea539a-ac7f-484d-adc3-c4e173460544 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:14.527481 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fea539a-ac7f-484d-adc3-c4e173460544 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:14.527755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fea539a-ac7f-484d-adc3-c4e173460544 admin admin] PATH_INFO: `/v3/projects/69bc01fd66694a188ec00074d35ea687/users/2b48ebda9b7648bebb4fa4712ab3c17c/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:14.529705 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fea539a-ac7f-484d-adc3-c4e173460544 admin admin] RBAC: Authorizing `identity:create_grant(project_id=69bc01fd66694a188ec00074d35ea687, user_id=2b48ebda9b7648bebb4fa4712ab3c17c, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:14.532365 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fea539a-ac7f-484d-adc3-c4e173460544 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:14.544665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 986/1967] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:14 2022] PUT /identity/v3/projects/69bc01fd66694a188ec00074d35ea687/users/2b48ebda9b7648bebb4fa4712ab3c17c/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:15.783444 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f23cd5b-1d24-421e-9e55-3a151000940f None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:15.783972 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f23cd5b-1d24-421e-9e55-3a151000940f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:15.784197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f23cd5b-1d24-421e-9e55-3a151000940f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:15.854815 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-0f23cd5b-1d24-421e-9e55-3a151000940f None None] MFA Rules not processed for user `2b48ebda9b7648bebb4fa4712ab3c17c`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:15.856425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-0f23cd5b-1d24-421e-9e55-3a151000940f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:16.008219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 982/1968] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:15 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:16.022120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fea539a-ac7f-484d-adc3-c4e173460544 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:16.024011 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-2fea539a-ac7f-484d-adc3-c4e173460544 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:16.148974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fea539a-ac7f-484d-adc3-c4e173460544 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:16.152872 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b2b6c4-eb92-4282-971f-b4776d9dce58 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:16.153633 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b2b6c4-eb92-4282-971f-b4776d9dce58 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:16.154102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b2b6c4-eb92-4282-971f-b4776d9dce58 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:16.154583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b2b6c4-eb92-4282-971f-b4776d9dce58 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:16.171121 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:16.277006 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b2b6c4-eb92-4282-971f-b4776d9dce58 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:16.278637 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b2b6c4-eb92-4282-971f-b4776d9dce58 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:16.298558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 987/1969] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:16 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 278 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:31:18.020513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f23cd5b-1d24-421e-9e55-3a151000940f None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:18.023361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-0f23cd5b-1d24-421e-9e55-3a151000940f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:18.044195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f23cd5b-1d24-421e-9e55-3a151000940f None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:18.048368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf2580c-0b25-4363-9896-2ea39aa3bf15 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:18.049301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf2580c-0b25-4363-9896-2ea39aa3bf15 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:18.049588 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf2580c-0b25-4363-9896-2ea39aa3bf15 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:18.049862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf2580c-0b25-4363-9896-2ea39aa3bf15 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:18.062150 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf2580c-0b25-4363-9896-2ea39aa3bf15 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:18.063832 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf2580c-0b25-4363-9896-2ea39aa3bf15 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:18.075384 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 983/1970] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:18 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:31:18.229776 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96704a66-5e5a-4203-b845-b280206242e6 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:18.230857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96704a66-5e5a-4203-b845-b280206242e6 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:18.231360 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96704a66-5e5a-4203-b845-b280206242e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:18.301512 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-96704a66-5e5a-4203-b845-b280206242e6 None None] MFA Rules not processed for user `c7028d2ccede4b44b567387bb83c4579`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:18.303003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-96704a66-5e5a-4203-b845-b280206242e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:18.392165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 988/1971] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:18 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:18.404149 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf2580c-0b25-4363-9896-2ea39aa3bf15 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:18.404149 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-cbf2580c-0b25-4363-9896-2ea39aa3bf15 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:18.466305 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:18.579702 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf2580c-0b25-4363-9896-2ea39aa3bf15 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:18.590908 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b07deb-4421-4bdf-94a5-3dcbee5bb5ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:18.592715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b07deb-4421-4bdf-94a5-3dcbee5bb5ca admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:18.599265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b07deb-4421-4bdf-94a5-3dcbee5bb5ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:18.599265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b07deb-4421-4bdf-94a5-3dcbee5bb5ca admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:18.599265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b07deb-4421-4bdf-94a5-3dcbee5bb5ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:18.603605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b07deb-4421-4bdf-94a5-3dcbee5bb5ca admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:18.659720 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 984/1972] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:31:18 2022] POST /identity/v3/projects => generated 379 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:18.669702 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96704a66-5e5a-4203-b845-b280206242e6 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:18.711440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96704a66-5e5a-4203-b845-b280206242e6 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:18.798143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2179757-a03a-4041-9075-8f7902fc9bd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:18.798839 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2179757-a03a-4041-9075-8f7902fc9bd4 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:18.799118 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2179757-a03a-4041-9075-8f7902fc9bd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:18.799453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2179757-a03a-4041-9075-8f7902fc9bd4 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:18.800119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2179757-a03a-4041-9075-8f7902fc9bd4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:18.802612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2179757-a03a-4041-9075-8f7902fc9bd4 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:18.840045 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 989/1973] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:31:18 2022] POST /identity/v3/users => generated 296 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:31:18.841561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b07deb-4421-4bdf-94a5-3dcbee5bb5ca admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:18.854566 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b07deb-4421-4bdf-94a5-3dcbee5bb5ca admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:18.860301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a1ee08-8571-4675-8ac8-e1b168c4a721 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:18.860963 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a1ee08-8571-4675-8ac8-e1b168c4a721 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:18.861243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a1ee08-8571-4675-8ac8-e1b168c4a721 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:18.861517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a1ee08-8571-4675-8ac8-e1b168c4a721 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:18.862181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a1ee08-8571-4675-8ac8-e1b168c4a721 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:18.864539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a1ee08-8571-4675-8ac8-e1b168c4a721 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:18.880375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 985/1974] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:18 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:18.883142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2179757-a03a-4041-9075-8f7902fc9bd4 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:18.897383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2179757-a03a-4041-9075-8f7902fc9bd4 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:18.906731 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7a9237-50c1-48dc-bb3c-d0f7c5d02029 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:18.909290 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7a9237-50c1-48dc-bb3c-d0f7c5d02029 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:18.910524 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7a9237-50c1-48dc-bb3c-d0f7c5d02029 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:18.911011 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7a9237-50c1-48dc-bb3c-d0f7c5d02029 admin admin] PATH_INFO: `/v3/projects/1a3633ae850147118ef0ca7e94bd5dc5/users/7f2b7f68137c404396c0e8f71dd6c5e6/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:18.938075 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7a9237-50c1-48dc-bb3c-d0f7c5d02029 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a3633ae850147118ef0ca7e94bd5dc5, user_id=7f2b7f68137c404396c0e8f71dd6c5e6, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:18.946205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7a9237-50c1-48dc-bb3c-d0f7c5d02029 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:18.970025 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 990/1975] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:18 2022] PUT /identity/v3/projects/1a3633ae850147118ef0ca7e94bd5dc5/users/7f2b7f68137c404396c0e8f71dd6c5e6/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:18.975840 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a1ee08-8571-4675-8ac8-e1b168c4a721 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:18.990760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a1ee08-8571-4675-8ac8-e1b168c4a721 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:19.085191 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717c12d2-ff9d-497f-bdb4-080e8454dbaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:19.087073 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717c12d2-ff9d-497f-bdb4-080e8454dbaf admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:19.087584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717c12d2-ff9d-497f-bdb4-080e8454dbaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:19.088041 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717c12d2-ff9d-497f-bdb4-080e8454dbaf admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:19.089028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717c12d2-ff9d-497f-bdb4-080e8454dbaf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:19.097283 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717c12d2-ff9d-497f-bdb4-080e8454dbaf admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:19.114771 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 986/1976] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:18 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:19.120558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7a9237-50c1-48dc-bb3c-d0f7c5d02029 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:19.142675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7a9237-50c1-48dc-bb3c-d0f7c5d02029 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:19.153883 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869b0e13-5ad0-4e93-8be8-2102a788e8a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:19.154589 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869b0e13-5ad0-4e93-8be8-2102a788e8a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:19.154877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869b0e13-5ad0-4e93-8be8-2102a788e8a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:19.155154 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869b0e13-5ad0-4e93-8be8-2102a788e8a1 admin admin] PATH_INFO: `/v3/projects/1a3633ae850147118ef0ca7e94bd5dc5/users/7f2b7f68137c404396c0e8f71dd6c5e6/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:19.158632 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869b0e13-5ad0-4e93-8be8-2102a788e8a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a3633ae850147118ef0ca7e94bd5dc5, user_id=7f2b7f68137c404396c0e8f71dd6c5e6, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:19.164578 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869b0e13-5ad0-4e93-8be8-2102a788e8a1 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:19.167816 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:19.192221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 991/1977] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:19 2022] PUT /identity/v3/projects/1a3633ae850147118ef0ca7e94bd5dc5/users/7f2b7f68137c404396c0e8f71dd6c5e6/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:31:27.433158 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b1952d-2ab5-4020-a2cf-491ed82be714 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:27.434033 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b1952d-2ab5-4020-a2cf-491ed82be714 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:27.434033 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b1952d-2ab5-4020-a2cf-491ed82be714 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:27.518731 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-21b1952d-2ab5-4020-a2cf-491ed82be714 None None] MFA Rules not processed for user `7f2b7f68137c404396c0e8f71dd6c5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:27.524481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-21b1952d-2ab5-4020-a2cf-491ed82be714 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:27.686689 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 987/1978] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:27 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:27.695000 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869b0e13-5ad0-4e93-8be8-2102a788e8a1 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:27.699257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-869b0e13-5ad0-4e93-8be8-2102a788e8a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:27.912852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869b0e13-5ad0-4e93-8be8-2102a788e8a1 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:27.927807 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652d79b5-30ee-404d-b8aa-756ce305ed17 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:27.930026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652d79b5-30ee-404d-b8aa-756ce305ed17 service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:27.930951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652d79b5-30ee-404d-b8aa-756ce305ed17 service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:27.932903 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652d79b5-30ee-404d-b8aa-756ce305ed17 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:28.054504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652d79b5-30ee-404d-b8aa-756ce305ed17 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:28.058203 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652d79b5-30ee-404d-b8aa-756ce305ed17 service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:28.092981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 992/1979] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:27 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:31:28.970861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3620f07-c5e7-4ba1-8fd3-ce0f057f6bc2 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:28.971966 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3620f07-c5e7-4ba1-8fd3-ce0f057f6bc2 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:28.972625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3620f07-c5e7-4ba1-8fd3-ce0f057f6bc2 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:28.974422 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 988/1980] 10.222.0.64 () {54 vars in 921 bytes} [Tue Oct 4 14:31:28 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 14:31:29.051964 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14541afd-8b99-4c29-b556-ab52ac9cb606 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:29.056162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14541afd-8b99-4c29-b556-ab52ac9cb606 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:29.056723 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14541afd-8b99-4c29-b556-ab52ac9cb606 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:29.117702 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-14541afd-8b99-4c29-b556-ab52ac9cb606 None None] MFA Rules not processed for user `436deb3d60c54a799700986519b2ef68`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:29.119139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-14541afd-8b99-4c29-b556-ab52ac9cb606 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:29.142320 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3620f07-c5e7-4ba1-8fd3-ce0f057f6bc2 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:29.146690 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-e3620f07-c5e7-4ba1-8fd3-ce0f057f6bc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:29.226091 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:29.269790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 993/1981] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Oct 4 14:31:29 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:29.302757 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466ebf1f-6639-4988-b6b6-d24472cc5d04 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:29.304475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466ebf1f-6639-4988-b6b6-d24472cc5d04 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:29.305704 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466ebf1f-6639-4988-b6b6-d24472cc5d04 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:29.308424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 994/1982] 10.222.0.64 () {54 vars in 961 bytes} [Tue Oct 4 14:31:29 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:31:29.314445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a961102-b268-4f46-8536-25cdff69ced5 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:29.315181 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a961102-b268-4f46-8536-25cdff69ced5 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:29.315678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a961102-b268-4f46-8536-25cdff69ced5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:29.338976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3620f07-c5e7-4ba1-8fd3-ce0f057f6bc2 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:29.346394 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603bdbca-ad69-4058-b9a0-365a3b5d6651 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:29.348129 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603bdbca-ad69-4058-b9a0-365a3b5d6651 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:29.348613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603bdbca-ad69-4058-b9a0-365a3b5d6651 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:29.349394 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603bdbca-ad69-4058-b9a0-365a3b5d6651 admin admin] PATH_INFO: `/v3/users/3f6d20541ee8417ba718190c88c48deb` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:29.351073 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603bdbca-ad69-4058-b9a0-365a3b5d6651 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3f6d20541ee8417ba718190c88c48deb)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:29.353650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603bdbca-ad69-4058-b9a0-365a3b5d6651 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:29.353876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-2a961102-b268-4f46-8536-25cdff69ced5 None None] MFA Rules not processed for user `436deb3d60c54a799700986519b2ef68`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:29.355093 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-2a961102-b268-4f46-8536-25cdff69ced5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:29.377322 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 995/1983] 10.222.0.64 () {58 vars in 1058 bytes} [Tue Oct 4 14:31:29 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:29.378929 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a961102-b268-4f46-8536-25cdff69ced5 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:29.404353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a961102-b268-4f46-8536-25cdff69ced5 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:29.415887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f40150-d1a9-45c3-b29e-0cf44473a6f1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '436deb3d60c54a799700986519b2ef68', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:29.416224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f40150-d1a9-45c3-b29e-0cf44473a6f1 service placement] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:29.416224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f40150-d1a9-45c3-b29e-0cf44473a6f1 service placement] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:29.416716 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f40150-d1a9-45c3-b29e-0cf44473a6f1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:29.427971 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f40150-d1a9-45c3-b29e-0cf44473a6f1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:29.429990 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f40150-d1a9-45c3-b29e-0cf44473a6f1 service placement] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:29.461407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 996/1984] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:29 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:31:29.526968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-603bdbca-ad69-4058-b9a0-365a3b5d6651 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3f6d20541ee8417ba718190c88c48deb'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:29.542419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-603bdbca-ad69-4058-b9a0-365a3b5d6651 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3f6d20541ee8417ba718190c88c48deb'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:29.573559 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-603bdbca-ad69-4058-b9a0-365a3b5d6651 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3f6d20541ee8417ba718190c88c48deb'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:29.594773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-603bdbca-ad69-4058-b9a0-365a3b5d6651 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3f6d20541ee8417ba718190c88c48deb'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:29.596687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-603bdbca-ad69-4058-b9a0-365a3b5d6651 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3f6d20541ee8417ba718190c88c48deb'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:29.628152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 989/1985] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:31:29 2022] DELETE /identity/v3/users/3f6d20541ee8417ba718190c88c48deb => generated 0 bytes in 485 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:29.976068 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f40150-d1a9-45c3-b29e-0cf44473a6f1 service placement] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:29.998776 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f40150-d1a9-45c3-b29e-0cf44473a6f1 service placement] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:30.004482 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:30.218007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f1797c-f462-4829-83b1-21b2a9a4a7d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:30.220299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f1797c-f462-4829-83b1-21b2a9a4a7d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:30.221043 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f1797c-f462-4829-83b1-21b2a9a4a7d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:30.221773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f1797c-f462-4829-83b1-21b2a9a4a7d7 admin admin] PATH_INFO: `/v3/projects/da10281be41c43988e60ecfafd5d0d12` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:30.229297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f1797c-f462-4829-83b1-21b2a9a4a7d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=da10281be41c43988e60ecfafd5d0d12)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:30.233978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f1797c-f462-4829-83b1-21b2a9a4a7d7 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:30.373665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e7f1797c-f462-4829-83b1-21b2a9a4a7d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'da10281be41c43988e60ecfafd5d0d12', 'request_id': 'req-e7f1797c-f462-4829-83b1-21b2a9a4a7d7'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:30.397710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e7f1797c-f462-4829-83b1-21b2a9a4a7d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'da10281be41c43988e60ecfafd5d0d12', 'request_id': 'req-e7f1797c-f462-4829-83b1-21b2a9a4a7d7'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:30.427128 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e7f1797c-f462-4829-83b1-21b2a9a4a7d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'da10281be41c43988e60ecfafd5d0d12', 'request_id': 'req-e7f1797c-f462-4829-83b1-21b2a9a4a7d7'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:30.483543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e7f1797c-f462-4829-83b1-21b2a9a4a7d7 admin admin] The token cache is being invalidate because project da10281be41c43988e60ecfafd5d0d12 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:31:30.483902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e7f1797c-f462-4829-83b1-21b2a9a4a7d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:30.489998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e7f1797c-f462-4829-83b1-21b2a9a4a7d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:30.493097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 997/1986] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:31:29 2022] DELETE /identity/v3/projects/da10281be41c43988e60ecfafd5d0d12 => generated 0 bytes in 518 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:30.501672 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980a5868-f03c-4506-b61f-baaf65e1bd87 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:30.503117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980a5868-f03c-4506-b61f-baaf65e1bd87 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:30.504202 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980a5868-f03c-4506-b61f-baaf65e1bd87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:30.573562 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-980a5868-f03c-4506-b61f-baaf65e1bd87 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:30.574853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-980a5868-f03c-4506-b61f-baaf65e1bd87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:30.791943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 990/1987] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:30.800284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ffa9f7-d011-4365-986f-96a2f1a6e23d None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:30.800809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ffa9f7-d011-4365-986f-96a2f1a6e23d None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:30.801304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ffa9f7-d011-4365-986f-96a2f1a6e23d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:30.863517 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-c7ffa9f7-d011-4365-986f-96a2f1a6e23d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:30.863517 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c7ffa9f7-d011-4365-986f-96a2f1a6e23d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:30.889500 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 998/1988] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:30.894663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980a5868-f03c-4506-b61f-baaf65e1bd87 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:30.914736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980a5868-f03c-4506-b61f-baaf65e1bd87 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:30.922869 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a99bbe8-098c-437b-a435-4537aa1c5f02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:30.923959 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a99bbe8-098c-437b-a435-4537aa1c5f02 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:30.924788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a99bbe8-098c-437b-a435-4537aa1c5f02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:30.926101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a99bbe8-098c-437b-a435-4537aa1c5f02 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:30.928807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a99bbe8-098c-437b-a435-4537aa1c5f02 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:31:30.929370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a99bbe8-098c-437b-a435-4537aa1c5f02 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:30.931456 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a99bbe8-098c-437b-a435-4537aa1c5f02 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:30.951266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 991/1989] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:31:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:31:30.958370 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ffa9f7-d011-4365-986f-96a2f1a6e23d None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:30.980880 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ffa9f7-d011-4365-986f-96a2f1a6e23d None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:30.987019 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3fe229-b29c-42aa-8bdc-3b4cf237ecec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:30.988399 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3fe229-b29c-42aa-8bdc-3b4cf237ecec admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:30.988806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3fe229-b29c-42aa-8bdc-3b4cf237ecec admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:30.989280 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3fe229-b29c-42aa-8bdc-3b4cf237ecec admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:30.991580 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3fe229-b29c-42aa-8bdc-3b4cf237ecec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:30.994089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3fe229-b29c-42aa-8bdc-3b4cf237ecec admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:31.060478 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 999/1990] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:31:30 2022] POST /identity/v3/projects => generated 405 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:31.062961 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a99bbe8-098c-437b-a435-4537aa1c5f02 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:31.087722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a99bbe8-098c-437b-a435-4537aa1c5f02 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:31.140202 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:31.232003 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd14305-88d2-4763-b9f3-32976b65174f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:31.232523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd14305-88d2-4763-b9f3-32976b65174f admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:31.232816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd14305-88d2-4763-b9f3-32976b65174f admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:31.233829 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd14305-88d2-4763-b9f3-32976b65174f admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:31.234496 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd14305-88d2-4763-b9f3-32976b65174f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:31.238501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd14305-88d2-4763-b9f3-32976b65174f admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:31.272467 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 992/1991] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:31:31 2022] POST /identity/v3/users => generated 303 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:31.278319 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3fe229-b29c-42aa-8bdc-3b4cf237ecec admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:31.304885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3fe229-b29c-42aa-8bdc-3b4cf237ecec admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:31.318790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e022f2ec-fed1-406e-ba44-8cc37cd19745 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:31.319755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e022f2ec-fed1-406e-ba44-8cc37cd19745 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:31.320165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e022f2ec-fed1-406e-ba44-8cc37cd19745 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:31.320538 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e022f2ec-fed1-406e-ba44-8cc37cd19745 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:31.322162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e022f2ec-fed1-406e-ba44-8cc37cd19745 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:31.324627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e022f2ec-fed1-406e-ba44-8cc37cd19745 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:31.343527 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1000/1992] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:31 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:31.347577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd14305-88d2-4763-b9f3-32976b65174f admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:31.359333 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd14305-88d2-4763-b9f3-32976b65174f admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:31.365217 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8d04f9-bd50-496a-a6af-e5c944b03e4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:31.365994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8d04f9-bd50-496a-a6af-e5c944b03e4f admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:31.366344 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8d04f9-bd50-496a-a6af-e5c944b03e4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:31.366714 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8d04f9-bd50-496a-a6af-e5c944b03e4f admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:31.367589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8d04f9-bd50-496a-a6af-e5c944b03e4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:31.369522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8d04f9-bd50-496a-a6af-e5c944b03e4f admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:31.387130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e022f2ec-fed1-406e-ba44-8cc37cd19745 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:31.387588 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 993/1993] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:31 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:31.404206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e022f2ec-fed1-406e-ba44-8cc37cd19745 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:31.417635 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb977acd-2366-4139-9185-972c817e3bbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:31.418963 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb977acd-2366-4139-9185-972c817e3bbd admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:31.419250 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb977acd-2366-4139-9185-972c817e3bbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:31.419654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb977acd-2366-4139-9185-972c817e3bbd admin admin] PATH_INFO: `/v3/projects/88ba00b9967746799e48089dd617626a/users/3f9de8414c4b4f37a8bdfd5308f6b447/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:31.462021 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb977acd-2366-4139-9185-972c817e3bbd admin admin] RBAC: Authorizing `identity:create_grant(project_id=88ba00b9967746799e48089dd617626a, user_id=3f9de8414c4b4f37a8bdfd5308f6b447, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:31.467732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb977acd-2366-4139-9185-972c817e3bbd admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:31.494508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1001/1994] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:31 2022] PUT /identity/v3/projects/88ba00b9967746799e48089dd617626a/users/3f9de8414c4b4f37a8bdfd5308f6b447/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:31:31.500838 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0469d4b8-b8a6-41b9-b6f2-01a005ce2f92 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:31.501779 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0469d4b8-b8a6-41b9-b6f2-01a005ce2f92 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:31.502380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0469d4b8-b8a6-41b9-b6f2-01a005ce2f92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:31.551154 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-0469d4b8-b8a6-41b9-b6f2-01a005ce2f92 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:31.553838 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-0469d4b8-b8a6-41b9-b6f2-01a005ce2f92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:31.699407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 994/1995] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:31.713284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb977acd-2366-4139-9185-972c817e3bbd admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:31.717756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-fb977acd-2366-4139-9185-972c817e3bbd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:31.815340 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:31.870113 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb977acd-2366-4139-9185-972c817e3bbd admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:31.875638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572fb6f8-38fa-4cfe-a51c-445ab6422b4e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:31.876396 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572fb6f8-38fa-4cfe-a51c-445ab6422b4e service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:31.876722 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572fb6f8-38fa-4cfe-a51c-445ab6422b4e service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:31.877057 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572fb6f8-38fa-4cfe-a51c-445ab6422b4e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:31.888960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572fb6f8-38fa-4cfe-a51c-445ab6422b4e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:31.890413 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572fb6f8-38fa-4cfe-a51c-445ab6422b4e service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:31.906942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1002/1996] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:31:33.258711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6588df-5df9-40d2-8fe7-f2c173e9b054 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:33.259922 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6588df-5df9-40d2-8fe7-f2c173e9b054 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:33.260504 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6588df-5df9-40d2-8fe7-f2c173e9b054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:33.315448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-7e6588df-5df9-40d2-8fe7-f2c173e9b054 None None] MFA Rules not processed for user `3f9de8414c4b4f37a8bdfd5308f6b447`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:33.317284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7e6588df-5df9-40d2-8fe7-f2c173e9b054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:33.437246 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2d2d45-f874-431b-b4ba-2a6d374c28f2 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:33.437768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2d2d45-f874-431b-b4ba-2a6d374c28f2 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:33.437893 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2d2d45-f874-431b-b4ba-2a6d374c28f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:33.447710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 995/1997] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:33 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:33.452860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6588df-5df9-40d2-8fe7-f2c173e9b054 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:33.464612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-ed2d2d45-f874-431b-b4ba-2a6d374c28f2 None None] MFA Rules not processed for user `b3bd51e1acb548ea957ffb57cb09f3f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:33.466374 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ed2d2d45-f874-431b-b4ba-2a6d374c28f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:33.475363 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6588df-5df9-40d2-8fe7-f2c173e9b054 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:33.481579 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a34c6a-6940-4e17-9b06-b278d5a1bf92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:33.482599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a34c6a-6940-4e17-9b06-b278d5a1bf92 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:33.483205 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a34c6a-6940-4e17-9b06-b278d5a1bf92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:33.483504 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a34c6a-6940-4e17-9b06-b278d5a1bf92 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:33.484175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a34c6a-6940-4e17-9b06-b278d5a1bf92 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:33.486447 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a34c6a-6940-4e17-9b06-b278d5a1bf92 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:33.528388 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 996/1998] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:31:33 2022] POST /identity/v3/projects => generated 405 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:33.533434 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a34c6a-6940-4e17-9b06-b278d5a1bf92 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:33.549652 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:33.558509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1003/1999] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:33 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:33.645037 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a34c6a-6940-4e17-9b06-b278d5a1bf92 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:33.650453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfe8736-1902-4607-bfbf-64fa435e804b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:33.651075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfe8736-1902-4607-bfbf-64fa435e804b admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:33.651432 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfe8736-1902-4607-bfbf-64fa435e804b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:33.651858 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfe8736-1902-4607-bfbf-64fa435e804b admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:33.652372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfe8736-1902-4607-bfbf-64fa435e804b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:33.654412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfe8736-1902-4607-bfbf-64fa435e804b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:33.681165 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 997/2000] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:31:33 2022] POST /identity/v3/users => generated 309 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:33.684809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2d2d45-f874-431b-b4ba-2a6d374c28f2 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:33.698943 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2d2d45-f874-431b-b4ba-2a6d374c28f2 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:33.704705 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f057a081-b2f0-4986-9d91-e697184d637a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:33.705335 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f057a081-b2f0-4986-9d91-e697184d637a admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:33.705612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f057a081-b2f0-4986-9d91-e697184d637a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:33.705882 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f057a081-b2f0-4986-9d91-e697184d637a admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:33.706613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f057a081-b2f0-4986-9d91-e697184d637a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:33.708485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f057a081-b2f0-4986-9d91-e697184d637a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:33.720810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1004/2001] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:33 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:33.725947 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfe8736-1902-4607-bfbf-64fa435e804b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:33.737185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfe8736-1902-4607-bfbf-64fa435e804b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:33.743692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feef6bb7-a5d0-4b17-8153-d742afa7b467 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:33.744527 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef6bb7-a5d0-4b17-8153-d742afa7b467 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:33.744836 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef6bb7-a5d0-4b17-8153-d742afa7b467 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:33.745106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef6bb7-a5d0-4b17-8153-d742afa7b467 admin admin] PATH_INFO: `/v3/projects/fab1cd71bf694e4f8faf416bad549998/users/8e87b57b22a344e0a6d1219c3c337f7f/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:33.763335 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feef6bb7-a5d0-4b17-8153-d742afa7b467 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fab1cd71bf694e4f8faf416bad549998, user_id=8e87b57b22a344e0a6d1219c3c337f7f, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:33.766080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feef6bb7-a5d0-4b17-8153-d742afa7b467 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:33.782032 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 998/2002] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:33 2022] PUT /identity/v3/projects/fab1cd71bf694e4f8faf416bad549998/users/8e87b57b22a344e0a6d1219c3c337f7f/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:31:33.789143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f057a081-b2f0-4986-9d91-e697184d637a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:33.807530 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f057a081-b2f0-4986-9d91-e697184d637a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:33.966879 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2371579e-0136-4ec6-b228-e8942ba1355a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:33.967924 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2371579e-0136-4ec6-b228-e8942ba1355a admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:33.968845 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2371579e-0136-4ec6-b228-e8942ba1355a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:33.969329 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2371579e-0136-4ec6-b228-e8942ba1355a admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:33.970752 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2371579e-0136-4ec6-b228-e8942ba1355a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:33.981861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2371579e-0136-4ec6-b228-e8942ba1355a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:34.010084 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1005/2003] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:33 2022] GET /identity/v3/roles => generated 1420 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:34.016341 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feef6bb7-a5d0-4b17-8153-d742afa7b467 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:34.040610 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feef6bb7-a5d0-4b17-8153-d742afa7b467 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:34.056142 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3a12f7-679d-4376-bf43-6661748b7ce3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:34.056980 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3a12f7-679d-4376-bf43-6661748b7ce3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:34.068782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3a12f7-679d-4376-bf43-6661748b7ce3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:34.069198 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3a12f7-679d-4376-bf43-6661748b7ce3 admin admin] PATH_INFO: `/v3/projects/fab1cd71bf694e4f8faf416bad549998/users/8e87b57b22a344e0a6d1219c3c337f7f/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:34.071648 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3a12f7-679d-4376-bf43-6661748b7ce3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fab1cd71bf694e4f8faf416bad549998, user_id=8e87b57b22a344e0a6d1219c3c337f7f, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:34.075503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3a12f7-679d-4376-bf43-6661748b7ce3 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:34.095959 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 999/2004] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:34 2022] PUT /identity/v3/projects/fab1cd71bf694e4f8faf416bad549998/users/8e87b57b22a344e0a6d1219c3c337f7f/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:35.549924 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a02eb43-5fa4-444a-af8b-970df90df854 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:35.552436 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a02eb43-5fa4-444a-af8b-970df90df854 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:35.553107 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a02eb43-5fa4-444a-af8b-970df90df854 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:35.619514 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-7a02eb43-5fa4-444a-af8b-970df90df854 None None] MFA Rules not processed for user `8e87b57b22a344e0a6d1219c3c337f7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:35.621409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7a02eb43-5fa4-444a-af8b-970df90df854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:35.720027 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:35.764972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1006/2005] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:35 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:36.197585 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782caed8-6919-4322-9c9c-139b59a30454 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:36.198051 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782caed8-6919-4322-9c9c-139b59a30454 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:36.198195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782caed8-6919-4322-9c9c-139b59a30454 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:36.200035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1000/2006] 10.222.0.64 () {54 vars in 921 bytes} [Tue Oct 4 14:31:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:31:36.226570 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb49984-18d1-4991-85d8-7b05a53a1945 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:36.227147 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb49984-18d1-4991-85d8-7b05a53a1945 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:36.227609 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb49984-18d1-4991-85d8-7b05a53a1945 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:36.250594 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-6eb49984-18d1-4991-85d8-7b05a53a1945 None None] MFA Rules not processed for user `436deb3d60c54a799700986519b2ef68`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:36.255814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-6eb49984-18d1-4991-85d8-7b05a53a1945 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:36.300466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782caed8-6919-4322-9c9c-139b59a30454 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:36.302490 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-782caed8-6919-4322-9c9c-139b59a30454 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:36.336900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1007/2007] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Oct 4 14:31:36 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:36.354120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb49984-18d1-4991-85d8-7b05a53a1945 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:36.355956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-6eb49984-18d1-4991-85d8-7b05a53a1945 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:36.371254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb49984-18d1-4991-85d8-7b05a53a1945 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:36.377326 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21e6e36-51d2-4e1d-bfe8-d3398b326ce8 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '436deb3d60c54a799700986519b2ef68', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:36.378332 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21e6e36-51d2-4e1d-bfe8-d3398b326ce8 service placement] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:36.378760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21e6e36-51d2-4e1d-bfe8-d3398b326ce8 service placement] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:36.379283 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21e6e36-51d2-4e1d-bfe8-d3398b326ce8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:36.397631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21e6e36-51d2-4e1d-bfe8-d3398b326ce8 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:36.399710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21e6e36-51d2-4e1d-bfe8-d3398b326ce8 service placement] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:36.423169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1008/2008] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:36 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:31:36.435070 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782caed8-6919-4322-9c9c-139b59a30454 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:36.440230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f41401d-7438-4594-a482-903b474ed0c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:36.441064 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f41401d-7438-4594-a482-903b474ed0c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:36.441342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f41401d-7438-4594-a482-903b474ed0c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:36.441609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f41401d-7438-4594-a482-903b474ed0c7 admin admin] PATH_INFO: `/v3/users/14df6dfb84e743268ff8cd5d7600c759` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:36.442949 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f41401d-7438-4594-a482-903b474ed0c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=14df6dfb84e743268ff8cd5d7600c759)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:36.444975 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f41401d-7438-4594-a482-903b474ed0c7 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:36.541402 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8f41401d-7438-4594-a482-903b474ed0c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '14df6dfb84e743268ff8cd5d7600c759'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:36.552358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8f41401d-7438-4594-a482-903b474ed0c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '14df6dfb84e743268ff8cd5d7600c759'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:36.561484 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8f41401d-7438-4594-a482-903b474ed0c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '14df6dfb84e743268ff8cd5d7600c759'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:36.563844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=86,keystone:INSERT=6,keystone:DELETE=8 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:36.593642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8f41401d-7438-4594-a482-903b474ed0c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '14df6dfb84e743268ff8cd5d7600c759'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:36.594301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8f41401d-7438-4594-a482-903b474ed0c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '14df6dfb84e743268ff8cd5d7600c759'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:36.602960 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1001/2009] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:31:36 2022] DELETE /identity/v3/users/14df6dfb84e743268ff8cd5d7600c759 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:36.933111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21e6e36-51d2-4e1d-bfe8-d3398b326ce8 service placement] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:36.950390 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21e6e36-51d2-4e1d-bfe8-d3398b326ce8 service placement] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:37.059663 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72f6d3c-703b-4195-a519-cc7d26dd40a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:37.060573 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f6d3c-703b-4195-a519-cc7d26dd40a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:37.060859 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f6d3c-703b-4195-a519-cc7d26dd40a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:37.061137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f6d3c-703b-4195-a519-cc7d26dd40a7 admin admin] PATH_INFO: `/v3/projects/29cbd816b9e24c17b3d0412e7cd85328` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:37.062956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72f6d3c-703b-4195-a519-cc7d26dd40a7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=29cbd816b9e24c17b3d0412e7cd85328)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:37.065088 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72f6d3c-703b-4195-a519-cc7d26dd40a7 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:37.125974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b72f6d3c-703b-4195-a519-cc7d26dd40a7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '29cbd816b9e24c17b3d0412e7cd85328', 'request_id': 'req-b72f6d3c-703b-4195-a519-cc7d26dd40a7'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:37.134492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b72f6d3c-703b-4195-a519-cc7d26dd40a7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '29cbd816b9e24c17b3d0412e7cd85328', 'request_id': 'req-b72f6d3c-703b-4195-a519-cc7d26dd40a7'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:37.142097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b72f6d3c-703b-4195-a519-cc7d26dd40a7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '29cbd816b9e24c17b3d0412e7cd85328', 'request_id': 'req-b72f6d3c-703b-4195-a519-cc7d26dd40a7'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:37.162388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b72f6d3c-703b-4195-a519-cc7d26dd40a7 admin admin] The token cache is being invalidate because project 29cbd816b9e24c17b3d0412e7cd85328 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:31:37.163227 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b72f6d3c-703b-4195-a519-cc7d26dd40a7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:37.164423 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b72f6d3c-703b-4195-a519-cc7d26dd40a7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:37.166355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1009/2010] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:31:36 2022] DELETE /identity/v3/projects/29cbd816b9e24c17b3d0412e7cd85328 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:37.176735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f41401d-7438-4594-a482-903b474ed0c7 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:37.179184 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-8f41401d-7438-4594-a482-903b474ed0c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:37.196551 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f41401d-7438-4594-a482-903b474ed0c7 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:37.308674 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ccf403-441c-433c-890c-01b8aab51e6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:37.309437 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ccf403-441c-433c-890c-01b8aab51e6d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:37.309923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ccf403-441c-433c-890c-01b8aab51e6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:37.311397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ccf403-441c-433c-890c-01b8aab51e6d admin admin] PATH_INFO: `/v3/users/2b48ebda9b7648bebb4fa4712ab3c17c` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:37.313228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ccf403-441c-433c-890c-01b8aab51e6d admin admin] RBAC: Authorizing `identity:delete_user(user_id=2b48ebda9b7648bebb4fa4712ab3c17c)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:37.318263 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ccf403-441c-433c-890c-01b8aab51e6d admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:37.413574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-d4ccf403-441c-433c-890c-01b8aab51e6d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2b48ebda9b7648bebb4fa4712ab3c17c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:37.429738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-d4ccf403-441c-433c-890c-01b8aab51e6d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2b48ebda9b7648bebb4fa4712ab3c17c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:37.446359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-d4ccf403-441c-433c-890c-01b8aab51e6d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2b48ebda9b7648bebb4fa4712ab3c17c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:37.470798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-d4ccf403-441c-433c-890c-01b8aab51e6d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2b48ebda9b7648bebb4fa4712ab3c17c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:37.473256 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-d4ccf403-441c-433c-890c-01b8aab51e6d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2b48ebda9b7648bebb4fa4712ab3c17c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:37.480357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1002/2011] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:31:37 2022] DELETE /identity/v3/users/2b48ebda9b7648bebb4fa4712ab3c17c => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:37.568925 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72f6d3c-703b-4195-a519-cc7d26dd40a7 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:37.571431 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-b72f6d3c-703b-4195-a519-cc7d26dd40a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:37.582124 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:37.588372 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72f6d3c-703b-4195-a519-cc7d26dd40a7 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:37.670132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3c38bf-5a01-4861-9e31-a7cc07b4c38f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:37.670781 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3c38bf-5a01-4861-9e31-a7cc07b4c38f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:37.671068 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3c38bf-5a01-4861-9e31-a7cc07b4c38f admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:37.671344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3c38bf-5a01-4861-9e31-a7cc07b4c38f admin admin] PATH_INFO: `/v3/users/3f9de8414c4b4f37a8bdfd5308f6b447` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:37.672538 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3c38bf-5a01-4861-9e31-a7cc07b4c38f admin admin] RBAC: Authorizing `identity:delete_user(user_id=3f9de8414c4b4f37a8bdfd5308f6b447)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:37.674471 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3c38bf-5a01-4861-9e31-a7cc07b4c38f admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:37.740975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0e3c38bf-5a01-4861-9e31-a7cc07b4c38f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3f9de8414c4b4f37a8bdfd5308f6b447'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:37.741836 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0e3c38bf-5a01-4861-9e31-a7cc07b4c38f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3f9de8414c4b4f37a8bdfd5308f6b447'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:37.742771 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0e3c38bf-5a01-4861-9e31-a7cc07b4c38f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3f9de8414c4b4f37a8bdfd5308f6b447'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:37.751810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0e3c38bf-5a01-4861-9e31-a7cc07b4c38f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3f9de8414c4b4f37a8bdfd5308f6b447'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:37.765724 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0e3c38bf-5a01-4861-9e31-a7cc07b4c38f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3f9de8414c4b4f37a8bdfd5308f6b447'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:37.779879 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1010/2012] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:31:37 2022] DELETE /identity/v3/users/3f9de8414c4b4f37a8bdfd5308f6b447 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:37.955978 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ccf403-441c-433c-890c-01b8aab51e6d admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:37.957904 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d4ccf403-441c-433c-890c-01b8aab51e6d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:37.973841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ccf403-441c-433c-890c-01b8aab51e6d admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:38.051569 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3c38bf-5a01-4861-9e31-a7cc07b4c38f admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:38.055404 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0e3c38bf-5a01-4861-9e31-a7cc07b4c38f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:38.112197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b854f00f-2427-4310-9b4c-b20928d27299 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:38.114250 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b854f00f-2427-4310-9b4c-b20928d27299 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:38.114653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b854f00f-2427-4310-9b4c-b20928d27299 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:38.114878 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b854f00f-2427-4310-9b4c-b20928d27299 admin admin] PATH_INFO: `/v3/projects/69bc01fd66694a188ec00074d35ea687` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:38.116096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b854f00f-2427-4310-9b4c-b20928d27299 admin admin] RBAC: Authorizing `identity:delete_project(project_id=69bc01fd66694a188ec00074d35ea687)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:38.118063 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b854f00f-2427-4310-9b4c-b20928d27299 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:38.120803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:38.183616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3c38bf-5a01-4861-9e31-a7cc07b4c38f admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:38.190073 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b854f00f-2427-4310-9b4c-b20928d27299 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '69bc01fd66694a188ec00074d35ea687', 'request_id': 'req-b854f00f-2427-4310-9b4c-b20928d27299'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:38.204197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b854f00f-2427-4310-9b4c-b20928d27299 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '69bc01fd66694a188ec00074d35ea687', 'request_id': 'req-b854f00f-2427-4310-9b4c-b20928d27299'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:38.215706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b854f00f-2427-4310-9b4c-b20928d27299 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '69bc01fd66694a188ec00074d35ea687', 'request_id': 'req-b854f00f-2427-4310-9b4c-b20928d27299'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:38.221052 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:38.235434 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b854f00f-2427-4310-9b4c-b20928d27299 admin admin] The token cache is being invalidate because project 69bc01fd66694a188ec00074d35ea687 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:31:38.235729 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b854f00f-2427-4310-9b4c-b20928d27299 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:38.236715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b854f00f-2427-4310-9b4c-b20928d27299 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:38.242463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1003/2013] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:31:37 2022] DELETE /identity/v3/projects/69bc01fd66694a188ec00074d35ea687 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:38.244163 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcd2f9b-1057-40ec-8b7a-4e7b88ae1449 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:38.244525 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcd2f9b-1057-40ec-8b7a-4e7b88ae1449 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:38.244777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcd2f9b-1057-40ec-8b7a-4e7b88ae1449 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:38.289390 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498f3621-5c86-477d-9cbb-bbc3c1f03a42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:38.289882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-afcd2f9b-1057-40ec-8b7a-4e7b88ae1449 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:38.290119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498f3621-5c86-477d-9cbb-bbc3c1f03a42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:38.290405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498f3621-5c86-477d-9cbb-bbc3c1f03a42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:38.290675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498f3621-5c86-477d-9cbb-bbc3c1f03a42 admin admin] PATH_INFO: `/v3/projects/88ba00b9967746799e48089dd617626a` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:38.291767 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-afcd2f9b-1057-40ec-8b7a-4e7b88ae1449 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:38.291901 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498f3621-5c86-477d-9cbb-bbc3c1f03a42 admin admin] RBAC: Authorizing `identity:delete_project(project_id=88ba00b9967746799e48089dd617626a)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:38.293966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498f3621-5c86-477d-9cbb-bbc3c1f03a42 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:38.301191 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1004/2014] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:38.310373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd8b2c4-4531-471e-a012-f58ce637daf3 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:38.311397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd8b2c4-4531-471e-a012-f58ce637daf3 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:38.312000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd8b2c4-4531-471e-a012-f58ce637daf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:38.342993 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-ebd8b2c4-4531-471e-a012-f58ce637daf3 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:38.344530 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ebd8b2c4-4531-471e-a012-f58ce637daf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:38.380612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-498f3621-5c86-477d-9cbb-bbc3c1f03a42 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '88ba00b9967746799e48089dd617626a', 'request_id': 'req-498f3621-5c86-477d-9cbb-bbc3c1f03a42'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:38.391730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-498f3621-5c86-477d-9cbb-bbc3c1f03a42 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '88ba00b9967746799e48089dd617626a', 'request_id': 'req-498f3621-5c86-477d-9cbb-bbc3c1f03a42'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:38.402452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-498f3621-5c86-477d-9cbb-bbc3c1f03a42 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '88ba00b9967746799e48089dd617626a', 'request_id': 'req-498f3621-5c86-477d-9cbb-bbc3c1f03a42'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:38.420397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-498f3621-5c86-477d-9cbb-bbc3c1f03a42 admin admin] The token cache is being invalidate because project 88ba00b9967746799e48089dd617626a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:31:38.420397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-498f3621-5c86-477d-9cbb-bbc3c1f03a42 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:38.420397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-498f3621-5c86-477d-9cbb-bbc3c1f03a42 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:38.425075 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1011/2015] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:31:38 2022] DELETE /identity/v3/projects/88ba00b9967746799e48089dd617626a => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:38.426979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498f3621-5c86-477d-9cbb-bbc3c1f03a42 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:38.428478 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-498f3621-5c86-477d-9cbb-bbc3c1f03a42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:38.470680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1005/2016] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:38.471561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd8b2c4-4531-471e-a012-f58ce637daf3 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:38.484800 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd8b2c4-4531-471e-a012-f58ce637daf3 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:38.491575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70ea87d-a133-490b-956b-7b25577bfb6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:38.492286 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70ea87d-a133-490b-956b-7b25577bfb6d admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:38.492603 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70ea87d-a133-490b-956b-7b25577bfb6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:38.493045 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70ea87d-a133-490b-956b-7b25577bfb6d admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:38.494460 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70ea87d-a133-490b-956b-7b25577bfb6d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:31:38.495074 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70ea87d-a133-490b-956b-7b25577bfb6d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:38.496597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70ea87d-a133-490b-956b-7b25577bfb6d admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:38.515950 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1006/2017] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:31:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:31:38.517265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70ea87d-a133-490b-956b-7b25577bfb6d admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:38.522798 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498f3621-5c86-477d-9cbb-bbc3c1f03a42 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:38.529313 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbb3ff0-2d0f-464e-8bb8-574f473a385b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:38.530110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbb3ff0-2d0f-464e-8bb8-574f473a385b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:38.530488 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbb3ff0-2d0f-464e-8bb8-574f473a385b admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:38.530866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbb3ff0-2d0f-464e-8bb8-574f473a385b admin admin] PATH_INFO: `/v3/users/8e87b57b22a344e0a6d1219c3c337f7f` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:38.531190 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70ea87d-a133-490b-956b-7b25577bfb6d admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:38.532201 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dbb3ff0-2d0f-464e-8bb8-574f473a385b admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e87b57b22a344e0a6d1219c3c337f7f)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:38.535042 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dbb3ff0-2d0f-464e-8bb8-574f473a385b admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:38.537043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801fa847-7c52-4836-b290-8c3a7670407e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:38.537674 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801fa847-7c52-4836-b290-8c3a7670407e admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:38.544533 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801fa847-7c52-4836-b290-8c3a7670407e admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:38.544533 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801fa847-7c52-4836-b290-8c3a7670407e admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:38.544533 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801fa847-7c52-4836-b290-8c3a7670407e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:38.544533 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801fa847-7c52-4836-b290-8c3a7670407e admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:38.585548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1007/2018] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:31:38 2022] POST /identity/v3/projects => generated 377 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:38.587052 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801fa847-7c52-4836-b290-8c3a7670407e admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:38.614201 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3dbb3ff0-2d0f-464e-8bb8-574f473a385b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e87b57b22a344e0a6d1219c3c337f7f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:38.615342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3dbb3ff0-2d0f-464e-8bb8-574f473a385b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e87b57b22a344e0a6d1219c3c337f7f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:38.616449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3dbb3ff0-2d0f-464e-8bb8-574f473a385b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e87b57b22a344e0a6d1219c3c337f7f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:38.631042 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3dbb3ff0-2d0f-464e-8bb8-574f473a385b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e87b57b22a344e0a6d1219c3c337f7f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:38.646805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:38.651814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3dbb3ff0-2d0f-464e-8bb8-574f473a385b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e87b57b22a344e0a6d1219c3c337f7f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:38.678693 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1012/2019] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:31:38 2022] DELETE /identity/v3/users/8e87b57b22a344e0a6d1219c3c337f7f => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:31:38.704539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:38.723945 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801fa847-7c52-4836-b290-8c3a7670407e admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:38.815810 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9ea0d0-7858-430c-85df-a1b73b2d8155 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:38.816775 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9ea0d0-7858-430c-85df-a1b73b2d8155 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:38.817062 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9ea0d0-7858-430c-85df-a1b73b2d8155 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:38.817554 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9ea0d0-7858-430c-85df-a1b73b2d8155 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:38.818314 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9ea0d0-7858-430c-85df-a1b73b2d8155 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:38.822477 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9ea0d0-7858-430c-85df-a1b73b2d8155 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:38.856440 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1008/2020] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:31:38 2022] POST /identity/v3/users => generated 289 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:38.860436 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbb3ff0-2d0f-464e-8bb8-574f473a385b admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:38.868344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-3dbb3ff0-2d0f-464e-8bb8-574f473a385b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:38.890381 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbb3ff0-2d0f-464e-8bb8-574f473a385b admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:38.900770 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f5fdc7-171e-4bd4-b0f6-66b7c6b4aab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:38.901671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f5fdc7-171e-4bd4-b0f6-66b7c6b4aab4 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:38.902179 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f5fdc7-171e-4bd4-b0f6-66b7c6b4aab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:38.902822 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f5fdc7-171e-4bd4-b0f6-66b7c6b4aab4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:38.904121 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f5fdc7-171e-4bd4-b0f6-66b7c6b4aab4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:38.906565 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f5fdc7-171e-4bd4-b0f6-66b7c6b4aab4 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:38.933513 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1013/2021] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:38 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:38.937473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9ea0d0-7858-430c-85df-a1b73b2d8155 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:38.949278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9ea0d0-7858-430c-85df-a1b73b2d8155 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:38.960419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4886c19-971b-4b70-9fbb-f244f2085de0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:38.961051 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4886c19-971b-4b70-9fbb-f244f2085de0 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:38.961330 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4886c19-971b-4b70-9fbb-f244f2085de0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:38.961602 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4886c19-971b-4b70-9fbb-f244f2085de0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:38.962254 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4886c19-971b-4b70-9fbb-f244f2085de0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:38.963866 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4886c19-971b-4b70-9fbb-f244f2085de0 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:38.978766 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1009/2022] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:38 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:38.980005 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f5fdc7-171e-4bd4-b0f6-66b7c6b4aab4 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:38.991383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f5fdc7-171e-4bd4-b0f6-66b7c6b4aab4 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:38.997487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d887e84d-a679-4828-9f7c-2181cc990070 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:38.998169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d887e84d-a679-4828-9f7c-2181cc990070 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:38.998454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d887e84d-a679-4828-9f7c-2181cc990070 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:38.998730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d887e84d-a679-4828-9f7c-2181cc990070 admin admin] PATH_INFO: `/v3/projects/280c40e0abb743e0bbeb98cd37b50065/users/eec0c3bc09ab4d8c86528bb21b6ba9c2/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.000675 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4886c19-971b-4b70-9fbb-f244f2085de0 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:39.003693 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d4886c19-971b-4b70-9fbb-f244f2085de0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:39.022181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4886c19-971b-4b70-9fbb-f244f2085de0 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:39.027053 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8182c884-b2ac-4a65-b76a-91b3e916c041 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:39.027712 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8182c884-b2ac-4a65-b76a-91b3e916c041 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:39.031270 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8182c884-b2ac-4a65-b76a-91b3e916c041 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:39.031547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8182c884-b2ac-4a65-b76a-91b3e916c041 admin admin] PATH_INFO: `/v3/projects/fab1cd71bf694e4f8faf416bad549998` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.035809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d887e84d-a679-4828-9f7c-2181cc990070 admin admin] RBAC: Authorizing `identity:create_grant(project_id=280c40e0abb743e0bbeb98cd37b50065, user_id=eec0c3bc09ab4d8c86528bb21b6ba9c2, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:39.038962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8182c884-b2ac-4a65-b76a-91b3e916c041 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fab1cd71bf694e4f8faf416bad549998)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:39.040937 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8182c884-b2ac-4a65-b76a-91b3e916c041 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:39.042509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d887e84d-a679-4828-9f7c-2181cc990070 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:39.060426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1014/2023] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:38 2022] PUT /identity/v3/projects/280c40e0abb743e0bbeb98cd37b50065/users/eec0c3bc09ab4d8c86528bb21b6ba9c2/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:39.071875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf16d0-45f0-4d94-a591-803e510a451c None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:39.072446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf16d0-45f0-4d94-a591-803e510a451c None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:39.073093 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf16d0-45f0-4d94-a591-803e510a451c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.107488 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8182c884-b2ac-4a65-b76a-91b3e916c041 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fab1cd71bf694e4f8faf416bad549998', 'request_id': 'req-8182c884-b2ac-4a65-b76a-91b3e916c041'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:39.116706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-0abf16d0-45f0-4d94-a591-803e510a451c None None] MFA Rules not processed for user `eec0c3bc09ab4d8c86528bb21b6ba9c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:39.117008 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8182c884-b2ac-4a65-b76a-91b3e916c041 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fab1cd71bf694e4f8faf416bad549998', 'request_id': 'req-8182c884-b2ac-4a65-b76a-91b3e916c041'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:39.117935 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0abf16d0-45f0-4d94-a591-803e510a451c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:39.123652 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8182c884-b2ac-4a65-b76a-91b3e916c041 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fab1cd71bf694e4f8faf416bad549998', 'request_id': 'req-8182c884-b2ac-4a65-b76a-91b3e916c041'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:39.139715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8182c884-b2ac-4a65-b76a-91b3e916c041 admin admin] The token cache is being invalidate because project fab1cd71bf694e4f8faf416bad549998 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:31:39.140012 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8182c884-b2ac-4a65-b76a-91b3e916c041 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:39.140733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8182c884-b2ac-4a65-b76a-91b3e916c041 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:39.142282 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1010/2024] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:31:38 2022] DELETE /identity/v3/projects/fab1cd71bf694e4f8faf416bad549998 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:31:39.147713 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491a2c34-aed5-45f1-8ebf-e82ef4cd6f36 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:39.147994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491a2c34-aed5-45f1-8ebf-e82ef4cd6f36 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:39.148261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491a2c34-aed5-45f1-8ebf-e82ef4cd6f36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.175284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-491a2c34-aed5-45f1-8ebf-e82ef4cd6f36 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:39.177172 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-491a2c34-aed5-45f1-8ebf-e82ef4cd6f36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:39.258462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1015/2025] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:39 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:39.258839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:39.264389 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abf16d0-45f0-4d94-a591-803e510a451c None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:39.266669 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0abf16d0-45f0-4d94-a591-803e510a451c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:39.274007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1011/2026] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:39.281872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6ff186-1574-476d-9d90-fb8f7ea8d9ba None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:39.282182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6ff186-1574-476d-9d90-fb8f7ea8d9ba None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:39.282691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6ff186-1574-476d-9d90-fb8f7ea8d9ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.301825 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abf16d0-45f0-4d94-a591-803e510a451c None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:39.306209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-ba6ff186-1574-476d-9d90-fb8f7ea8d9ba None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:39.307422 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ba6ff186-1574-476d-9d90-fb8f7ea8d9ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:39.314636 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e27c710-05db-4e0a-963d-c5f1003c909e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:39.315194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e27c710-05db-4e0a-963d-c5f1003c909e admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:39.315755 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e27c710-05db-4e0a-963d-c5f1003c909e admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:39.316197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e27c710-05db-4e0a-963d-c5f1003c909e admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.316893 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e27c710-05db-4e0a-963d-c5f1003c909e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:39.318821 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1012/2027] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:39.322880 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6ff186-1574-476d-9d90-fb8f7ea8d9ba None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:39.327108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e27c710-05db-4e0a-963d-c5f1003c909e admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:39.333997 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6ff186-1574-476d-9d90-fb8f7ea8d9ba None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:39.342388 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507ef061-d478-4ce6-961f-6a22c32b9e92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:39.343006 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507ef061-d478-4ce6-961f-6a22c32b9e92 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:39.343285 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507ef061-d478-4ce6-961f-6a22c32b9e92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:39.343556 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507ef061-d478-4ce6-961f-6a22c32b9e92 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.344214 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507ef061-d478-4ce6-961f-6a22c32b9e92 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:31:39.344552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507ef061-d478-4ce6-961f-6a22c32b9e92 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:39.346191 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507ef061-d478-4ce6-961f-6a22c32b9e92 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:39.364672 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1013/2028] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:31:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:31:39.367498 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507ef061-d478-4ce6-961f-6a22c32b9e92 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:39.389207 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507ef061-d478-4ce6-961f-6a22c32b9e92 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:39.395674 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f42f63f-715d-48b2-ab59-0afe6f597a29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:39.396407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f42f63f-715d-48b2-ab59-0afe6f597a29 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:39.396687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f42f63f-715d-48b2-ab59-0afe6f597a29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:39.396961 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f42f63f-715d-48b2-ab59-0afe6f597a29 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.397594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f42f63f-715d-48b2-ab59-0afe6f597a29 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:39.399501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f42f63f-715d-48b2-ab59-0afe6f597a29 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:39.399898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1016/2029] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:31:39 2022] POST /identity/v3/projects => generated 375 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:31:39.403892 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e27c710-05db-4e0a-963d-c5f1003c909e admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:39.434535 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1014/2030] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:31:39 2022] POST /identity/v3/projects => generated 363 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:31:39.436599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f42f63f-715d-48b2-ab59-0afe6f597a29 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:39.498171 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e27c710-05db-4e0a-963d-c5f1003c909e admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:39.519231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:DELETE=1,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:39.542059 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f42f63f-715d-48b2-ab59-0afe6f597a29 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:39.549913 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53a48c4-9892-4431-87d3-81dd2f516df0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:39.550994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53a48c4-9892-4431-87d3-81dd2f516df0 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:39.551588 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53a48c4-9892-4431-87d3-81dd2f516df0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:39.552171 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53a48c4-9892-4431-87d3-81dd2f516df0 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.553507 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53a48c4-9892-4431-87d3-81dd2f516df0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:39.557348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53a48c4-9892-4431-87d3-81dd2f516df0 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:39.599727 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1015/2031] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:31:39 2022] POST /identity/v3/users => generated 282 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:39.604185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53a48c4-9892-4431-87d3-81dd2f516df0 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:39.618897 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc211d1-9e4c-4151-9ac4-3a2607c6dfc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:39.619090 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53a48c4-9892-4431-87d3-81dd2f516df0 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:39.620246 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc211d1-9e4c-4151-9ac4-3a2607c6dfc5 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:39.620582 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc211d1-9e4c-4151-9ac4-3a2607c6dfc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:39.624248 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc211d1-9e4c-4151-9ac4-3a2607c6dfc5 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.624248 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc211d1-9e4c-4151-9ac4-3a2607c6dfc5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:39.624670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc211d1-9e4c-4151-9ac4-3a2607c6dfc5 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:39.630331 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb0cb4f-6ad2-4a01-8e86-01b531171fb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:39.631113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb0cb4f-6ad2-4a01-8e86-01b531171fb9 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:39.631472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb0cb4f-6ad2-4a01-8e86-01b531171fb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:39.631824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb0cb4f-6ad2-4a01-8e86-01b531171fb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.632595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb0cb4f-6ad2-4a01-8e86-01b531171fb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:39.634291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb0cb4f-6ad2-4a01-8e86-01b531171fb9 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:39.648225 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1016/2032] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:39 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:39.652366 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb0cb4f-6ad2-4a01-8e86-01b531171fb9 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:39.664048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1017/2033] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:31:39 2022] POST /identity/v3/users => generated 288 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:31:39.666285 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb0cb4f-6ad2-4a01-8e86-01b531171fb9 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:39.668290 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc211d1-9e4c-4151-9ac4-3a2607c6dfc5 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:39.681316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dba48da-ccee-4470-aed9-1a12d292cfb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:39.682019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dba48da-ccee-4470-aed9-1a12d292cfb7 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:39.682301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dba48da-ccee-4470-aed9-1a12d292cfb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:39.682630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dba48da-ccee-4470-aed9-1a12d292cfb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.683841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dba48da-ccee-4470-aed9-1a12d292cfb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:39.686091 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dba48da-ccee-4470-aed9-1a12d292cfb7 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:39.691904 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc211d1-9e4c-4151-9ac4-3a2607c6dfc5 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:39.703936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1017/2034] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:39 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:31:39.705193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dba48da-ccee-4470-aed9-1a12d292cfb7 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:39.707989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae730fc-ecbf-44b5-bb27-46b5240a49f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:39.708661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae730fc-ecbf-44b5-bb27-46b5240a49f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:39.709486 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae730fc-ecbf-44b5-bb27-46b5240a49f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:39.709486 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae730fc-ecbf-44b5-bb27-46b5240a49f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.709852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae730fc-ecbf-44b5-bb27-46b5240a49f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:39.711661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae730fc-ecbf-44b5-bb27-46b5240a49f5 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:39.717097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dba48da-ccee-4470-aed9-1a12d292cfb7 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:39.725658 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1018/2035] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:39 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:39.726099 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10def966-1b09-47cf-b9cb-09205f446e4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:39.727124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10def966-1b09-47cf-b9cb-09205f446e4b admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:39.727289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10def966-1b09-47cf-b9cb-09205f446e4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:39.727565 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10def966-1b09-47cf-b9cb-09205f446e4b admin admin] PATH_INFO: `/v3/projects/9cb8da2b8315493387c4c340b31fee37/users/137b4bf19b664f5fa64fbdf9df7b6536/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.730212 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae730fc-ecbf-44b5-bb27-46b5240a49f5 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:39.745388 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10def966-1b09-47cf-b9cb-09205f446e4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=9cb8da2b8315493387c4c340b31fee37, user_id=137b4bf19b664f5fa64fbdf9df7b6536, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:39.747951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae730fc-ecbf-44b5-bb27-46b5240a49f5 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:39.749035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10def966-1b09-47cf-b9cb-09205f446e4b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:39.754607 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98ac933-9003-47d4-9ffe-003f640e6439 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:39.755529 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98ac933-9003-47d4-9ffe-003f640e6439 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:39.755826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98ac933-9003-47d4-9ffe-003f640e6439 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:39.756136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98ac933-9003-47d4-9ffe-003f640e6439 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.756856 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98ac933-9003-47d4-9ffe-003f640e6439 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:39.758801 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98ac933-9003-47d4-9ffe-003f640e6439 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:39.765916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1018/2036] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:39 2022] PUT /identity/v3/projects/9cb8da2b8315493387c4c340b31fee37/users/137b4bf19b664f5fa64fbdf9df7b6536/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:39.779976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ee5405-bd30-40ff-a29a-51312092c00a None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:39.780632 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ee5405-bd30-40ff-a29a-51312092c00a None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:39.780930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ee5405-bd30-40ff-a29a-51312092c00a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.781858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1019/2037] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:39 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:31:39.786264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98ac933-9003-47d4-9ffe-003f640e6439 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:39.798385 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98ac933-9003-47d4-9ffe-003f640e6439 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:39.826163 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-37ee5405-bd30-40ff-a29a-51312092c00a None None] MFA Rules not processed for user `137b4bf19b664f5fa64fbdf9df7b6536`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:39.827931 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-37ee5405-bd30-40ff-a29a-51312092c00a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:39.879190 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3863e8-d638-47c4-972e-7999d7230d1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:39.879881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3863e8-d638-47c4-972e-7999d7230d1b admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:39.880170 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3863e8-d638-47c4-972e-7999d7230d1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:39.880480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3863e8-d638-47c4-972e-7999d7230d1b admin admin] PATH_INFO: `/v3/projects/644f0263851d4b90917847e9c3c0b8cc/users/07e152b1680a4121b8e9d4cb59f6560b/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.897001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:39.900220 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3863e8-d638-47c4-972e-7999d7230d1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=644f0263851d4b90917847e9c3c0b8cc, user_id=07e152b1680a4121b8e9d4cb59f6560b, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:39.903824 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3863e8-d638-47c4-972e-7999d7230d1b admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:39.919525 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1019/2038] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:39 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:39.921449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1020/2039] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:39 2022] PUT /identity/v3/projects/644f0263851d4b90917847e9c3c0b8cc/users/07e152b1680a4121b8e9d4cb59f6560b/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:31:39.925772 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ee5405-bd30-40ff-a29a-51312092c00a None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:39.931281 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e2f7c6-f335-4e0e-a5aa-8a9969fde4ea None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:39.931759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e2f7c6-f335-4e0e-a5aa-8a9969fde4ea None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:39.932043 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e2f7c6-f335-4e0e-a5aa-8a9969fde4ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:39.937240 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ee5405-bd30-40ff-a29a-51312092c00a None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:39.978339 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-c9e2f7c6-f335-4e0e-a5aa-8a9969fde4ea None None] MFA Rules not processed for user `07e152b1680a4121b8e9d4cb59f6560b`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:39.979489 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c9e2f7c6-f335-4e0e-a5aa-8a9969fde4ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:40.038574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8951501-4193-4dbc-95f6-98fd91d1b994 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:40.038574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8951501-4193-4dbc-95f6-98fd91d1b994 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:40.038574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8951501-4193-4dbc-95f6-98fd91d1b994 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:40.038574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8951501-4193-4dbc-95f6-98fd91d1b994 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:40.039046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8951501-4193-4dbc-95f6-98fd91d1b994 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:40.039046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8951501-4193-4dbc-95f6-98fd91d1b994 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:40.072852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:40.076572 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1021/2040] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:39 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:40.081617 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e2f7c6-f335-4e0e-a5aa-8a9969fde4ea None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:40.083575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1020/2041] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:31:39 2022] POST /identity/v3/projects => generated 363 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:31:40.084530 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8951501-4193-4dbc-95f6-98fd91d1b994 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:40.099610 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e2f7c6-f335-4e0e-a5aa-8a9969fde4ea None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:40.172508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8951501-4193-4dbc-95f6-98fd91d1b994 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:40.178555 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f530fbdb-de07-4abb-b2c3-f8a1ef81a4e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:40.179499 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f530fbdb-de07-4abb-b2c3-f8a1ef81a4e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:40.179499 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f530fbdb-de07-4abb-b2c3-f8a1ef81a4e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:40.179755 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f530fbdb-de07-4abb-b2c3-f8a1ef81a4e9 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:40.180304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f530fbdb-de07-4abb-b2c3-f8a1ef81a4e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:40.182256 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f530fbdb-de07-4abb-b2c3-f8a1ef81a4e9 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:40.195058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68f5d0a-edb4-4469-b8b6-86c8003f0bbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:40.195773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68f5d0a-edb4-4469-b8b6-86c8003f0bbb admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:40.196131 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68f5d0a-edb4-4469-b8b6-86c8003f0bbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:40.197022 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68f5d0a-edb4-4469-b8b6-86c8003f0bbb admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:40.198426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68f5d0a-edb4-4469-b8b6-86c8003f0bbb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:40.199567 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68f5d0a-edb4-4469-b8b6-86c8003f0bbb admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:40.217341 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1021/2042] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:31:40 2022] POST /identity/v3/users => generated 288 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:40.221825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f530fbdb-de07-4abb-b2c3-f8a1ef81a4e9 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:40.233400 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f530fbdb-de07-4abb-b2c3-f8a1ef81a4e9 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:40.239073 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebb7f92-4347-4d59-91da-baeeebf42217 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:40.239594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebb7f92-4347-4d59-91da-baeeebf42217 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:40.240017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebb7f92-4347-4d59-91da-baeeebf42217 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:40.240285 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebb7f92-4347-4d59-91da-baeeebf42217 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:40.241020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aebb7f92-4347-4d59-91da-baeeebf42217 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:40.242435 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1022/2043] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:31:40 2022] POST /identity/v3/projects => generated 377 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:40.242611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aebb7f92-4347-4d59-91da-baeeebf42217 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:40.246521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68f5d0a-edb4-4469-b8b6-86c8003f0bbb admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:40.255656 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1022/2044] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:40 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:31:40.259682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebb7f92-4347-4d59-91da-baeeebf42217 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:40.274462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebb7f92-4347-4d59-91da-baeeebf42217 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:40.341105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68f5d0a-edb4-4469-b8b6-86c8003f0bbb admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:40.347516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4aada7-d94a-4efd-99d7-b35550e3403c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:40.348259 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4aada7-d94a-4efd-99d7-b35550e3403c admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:40.348583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4aada7-d94a-4efd-99d7-b35550e3403c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:40.349063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4aada7-d94a-4efd-99d7-b35550e3403c admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:40.349909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4aada7-d94a-4efd-99d7-b35550e3403c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:40.351065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:40.351818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4aada7-d94a-4efd-99d7-b35550e3403c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:40.365826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c42b4f-6143-4bb4-bffc-dd3f6ba9c5f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:40.366979 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c42b4f-6143-4bb4-bffc-dd3f6ba9c5f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:40.367491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c42b4f-6143-4bb4-bffc-dd3f6ba9c5f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:40.367997 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c42b4f-6143-4bb4-bffc-dd3f6ba9c5f7 admin admin] PATH_INFO: `/v3/projects/ac05002c2603474394372c9e94b5e5c0/users/74117a5146774d17b355fa590c0f3e44/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:40.377435 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1023/2045] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:31:40 2022] POST /identity/v3/users => generated 295 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:31:40.385810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4aada7-d94a-4efd-99d7-b35550e3403c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:40.399654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4aada7-d94a-4efd-99d7-b35550e3403c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:40.400092 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c42b4f-6143-4bb4-bffc-dd3f6ba9c5f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac05002c2603474394372c9e94b5e5c0, user_id=74117a5146774d17b355fa590c0f3e44, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:40.404443 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c42b4f-6143-4bb4-bffc-dd3f6ba9c5f7 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:40.406562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72d98c0-7a2c-4263-9700-c1a7cf69cdd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:40.407537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72d98c0-7a2c-4263-9700-c1a7cf69cdd1 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:40.407910 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72d98c0-7a2c-4263-9700-c1a7cf69cdd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:40.408435 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72d98c0-7a2c-4263-9700-c1a7cf69cdd1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:40.410121 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72d98c0-7a2c-4263-9700-c1a7cf69cdd1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:40.412916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72d98c0-7a2c-4263-9700-c1a7cf69cdd1 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:40.425969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1023/2046] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:40 2022] PUT /identity/v3/projects/ac05002c2603474394372c9e94b5e5c0/users/74117a5146774d17b355fa590c0f3e44/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:31:40.432430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c42b4f-6143-4bb4-bffc-dd3f6ba9c5f7 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:40.436145 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1024/2047] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:40 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:40.437921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72d98c0-7a2c-4263-9700-c1a7cf69cdd1 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:40.451188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72d98c0-7a2c-4263-9700-c1a7cf69cdd1 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:40.543369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcba2b2b-05e5-450a-9c20-a0d75fd7f446 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:40.544573 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcba2b2b-05e5-450a-9c20-a0d75fd7f446 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:40.545051 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcba2b2b-05e5-450a-9c20-a0d75fd7f446 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:40.549047 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcba2b2b-05e5-450a-9c20-a0d75fd7f446 admin admin] PATH_INFO: `/v3/projects/b4b5cb0a3b704fbd9ae1bfea14b55a6e/users/d37a20c847eb4d599b8ddb4822373c13/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:40.561757 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:40.570544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c42b4f-6143-4bb4-bffc-dd3f6ba9c5f7 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:40.574658 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcba2b2b-05e5-450a-9c20-a0d75fd7f446 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b4b5cb0a3b704fbd9ae1bfea14b55a6e, user_id=d37a20c847eb4d599b8ddb4822373c13, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:40.575673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff2cdb7-68d2-4e0f-9efe-5ac8f6e650c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:40.576298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff2cdb7-68d2-4e0f-9efe-5ac8f6e650c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:40.576607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff2cdb7-68d2-4e0f-9efe-5ac8f6e650c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:40.576880 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff2cdb7-68d2-4e0f-9efe-5ac8f6e650c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:40.577526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff2cdb7-68d2-4e0f-9efe-5ac8f6e650c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:40.578700 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcba2b2b-05e5-450a-9c20-a0d75fd7f446 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:40.579117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff2cdb7-68d2-4e0f-9efe-5ac8f6e650c9 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:40.590894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1024/2048] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:40 2022] GET /identity/v3/roles => generated 1420 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:40.594777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff2cdb7-68d2-4e0f-9efe-5ac8f6e650c9 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:40.595634 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1025/2049] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:40 2022] PUT /identity/v3/projects/b4b5cb0a3b704fbd9ae1bfea14b55a6e/users/d37a20c847eb4d599b8ddb4822373c13/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:40.601423 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcba2b2b-05e5-450a-9c20-a0d75fd7f446 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:40.610089 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff2cdb7-68d2-4e0f-9efe-5ac8f6e650c9 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:40.681717 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcba2b2b-05e5-450a-9c20-a0d75fd7f446 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:40.686781 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c54a73d-34b3-42a4-827b-b948a7b1cb37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:40.687400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c54a73d-34b3-42a4-827b-b948a7b1cb37 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:40.687687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c54a73d-34b3-42a4-827b-b948a7b1cb37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:40.687957 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c54a73d-34b3-42a4-827b-b948a7b1cb37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:40.688629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c54a73d-34b3-42a4-827b-b948a7b1cb37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:40.690225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c54a73d-34b3-42a4-827b-b948a7b1cb37 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:40.700568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8cd6c8-5526-458a-85b7-6af0171dd290 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:40.701355 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8cd6c8-5526-458a-85b7-6af0171dd290 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:40.702446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8cd6c8-5526-458a-85b7-6af0171dd290 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:40.702446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8cd6c8-5526-458a-85b7-6af0171dd290 admin admin] PATH_INFO: `/v3/projects/ac05002c2603474394372c9e94b5e5c0/users/74117a5146774d17b355fa590c0f3e44/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:40.702699 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1026/2050] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:40 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:31:40.704194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8cd6c8-5526-458a-85b7-6af0171dd290 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac05002c2603474394372c9e94b5e5c0, user_id=74117a5146774d17b355fa590c0f3e44, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:40.707506 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c54a73d-34b3-42a4-827b-b948a7b1cb37 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:40.708115 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8cd6c8-5526-458a-85b7-6af0171dd290 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:40.718335 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c54a73d-34b3-42a4-827b-b948a7b1cb37 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:40.720165 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1025/2051] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:40 2022] PUT /identity/v3/projects/ac05002c2603474394372c9e94b5e5c0/users/74117a5146774d17b355fa590c0f3e44/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:31:40.731298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b32781-0b82-473d-8ac5-68feee2ef365 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:40.731589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b32781-0b82-473d-8ac5-68feee2ef365 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:40.731872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b32781-0b82-473d-8ac5-68feee2ef365 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:40.766383 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-80b32781-0b82-473d-8ac5-68feee2ef365 None None] MFA Rules not processed for user `74117a5146774d17b355fa590c0f3e44`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:40.767709 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-80b32781-0b82-473d-8ac5-68feee2ef365 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:40.792366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3e4fdd-5592-4281-95ea-0fb6eccb91ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:40.793282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3e4fdd-5592-4281-95ea-0fb6eccb91ab admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:40.793567 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3e4fdd-5592-4281-95ea-0fb6eccb91ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:40.793882 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3e4fdd-5592-4281-95ea-0fb6eccb91ab admin admin] PATH_INFO: `/v3/projects/b4b5cb0a3b704fbd9ae1bfea14b55a6e/users/d37a20c847eb4d599b8ddb4822373c13/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:40.795905 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3e4fdd-5592-4281-95ea-0fb6eccb91ab admin admin] RBAC: Authorizing `identity:create_grant(project_id=b4b5cb0a3b704fbd9ae1bfea14b55a6e, user_id=d37a20c847eb4d599b8ddb4822373c13, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:40.798693 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3e4fdd-5592-4281-95ea-0fb6eccb91ab admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:40.811919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1027/2052] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:40 2022] PUT /identity/v3/projects/b4b5cb0a3b704fbd9ae1bfea14b55a6e/users/d37a20c847eb4d599b8ddb4822373c13/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:40.824231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48839240-d26f-4196-b1d5-11d575884f57 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:40.824562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48839240-d26f-4196-b1d5-11d575884f57 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:40.824808 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48839240-d26f-4196-b1d5-11d575884f57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:40.838502 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:40.850965 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1026/2053] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:40 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:40.854152 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-48839240-d26f-4196-b1d5-11d575884f57 None None] MFA Rules not processed for user `d37a20c847eb4d599b8ddb4822373c13`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:40.855206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-48839240-d26f-4196-b1d5-11d575884f57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:40.862880 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b32781-0b82-473d-8ac5-68feee2ef365 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:40.864666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-80b32781-0b82-473d-8ac5-68feee2ef365 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:40.930632 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b32781-0b82-473d-8ac5-68feee2ef365 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:40.941489 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1028/2054] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:40 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:40.954279 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48839240-d26f-4196-b1d5-11d575884f57 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:40.955628 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-48839240-d26f-4196-b1d5-11d575884f57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:40.961361 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:40.986167 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0bfbbc-96ac-41a4-b088-98d2ffe59c25 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:40.986795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0bfbbc-96ac-41a4-b088-98d2ffe59c25 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:40.987076 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0bfbbc-96ac-41a4-b088-98d2ffe59c25 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:40.988133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0bfbbc-96ac-41a4-b088-98d2ffe59c25 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:41.042243 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48839240-d26f-4196-b1d5-11d575884f57 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:41.046218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39a662f-5726-47e4-9485-bb0a54ba8697 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:41.046861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39a662f-5726-47e4-9485-bb0a54ba8697 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:41.047397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39a662f-5726-47e4-9485-bb0a54ba8697 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:41.047397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39a662f-5726-47e4-9485-bb0a54ba8697 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:41.064554 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0bfbbc-96ac-41a4-b088-98d2ffe59c25 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:41.066621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0bfbbc-96ac-41a4-b088-98d2ffe59c25 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:41.082544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1027/2055] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:40 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:31:41.127573 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39a662f-5726-47e4-9485-bb0a54ba8697 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:41.129300 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39a662f-5726-47e4-9485-bb0a54ba8697 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:41.144082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1029/2056] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:40 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:31:41.737265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0bfbbc-96ac-41a4-b088-98d2ffe59c25 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:41.806663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0bfbbc-96ac-41a4-b088-98d2ffe59c25 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:41.812122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bc2446-8f56-4eef-98a7-d03510226336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:41.812783 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bc2446-8f56-4eef-98a7-d03510226336 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:41.813053 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bc2446-8f56-4eef-98a7-d03510226336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:41.813324 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bc2446-8f56-4eef-98a7-d03510226336 admin admin] PATH_INFO: `/v3/users/137b4bf19b664f5fa64fbdf9df7b6536` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:41.814436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bc2446-8f56-4eef-98a7-d03510226336 admin admin] RBAC: Authorizing `identity:delete_user(user_id=137b4bf19b664f5fa64fbdf9df7b6536)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:41.817272 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bc2446-8f56-4eef-98a7-d03510226336 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:41.818997 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:41.882997 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-f5bc2446-8f56-4eef-98a7-d03510226336 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '137b4bf19b664f5fa64fbdf9df7b6536'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:41.891017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-f5bc2446-8f56-4eef-98a7-d03510226336 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '137b4bf19b664f5fa64fbdf9df7b6536'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:41.900876 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-f5bc2446-8f56-4eef-98a7-d03510226336 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '137b4bf19b664f5fa64fbdf9df7b6536'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:41.921075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-f5bc2446-8f56-4eef-98a7-d03510226336 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '137b4bf19b664f5fa64fbdf9df7b6536'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:41.921850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-f5bc2446-8f56-4eef-98a7-d03510226336 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '137b4bf19b664f5fa64fbdf9df7b6536'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:41.930441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1028/2057] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:31:41 2022] DELETE /identity/v3/users/137b4bf19b664f5fa64fbdf9df7b6536 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:41.936244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447d51e4-26c0-4ef1-ac85-c32b375eb19d None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:41.936505 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447d51e4-26c0-4ef1-ac85-c32b375eb19d None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:41.936772 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447d51e4-26c0-4ef1-ac85-c32b375eb19d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:41.957530 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-447d51e4-26c0-4ef1-ac85-c32b375eb19d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:41.958936 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-447d51e4-26c0-4ef1-ac85-c32b375eb19d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:42.064243 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1030/2058] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:42.072438 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bc2446-8f56-4eef-98a7-d03510226336 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:42.074145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f5bc2446-8f56-4eef-98a7-d03510226336 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:42.151553 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bc2446-8f56-4eef-98a7-d03510226336 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:42.156258 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6efb12e-295e-4f90-bc32-395949e0b9a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:42.156936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6efb12e-295e-4f90-bc32-395949e0b9a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:42.157208 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6efb12e-295e-4f90-bc32-395949e0b9a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:42.157731 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6efb12e-295e-4f90-bc32-395949e0b9a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:42.168920 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6efb12e-295e-4f90-bc32-395949e0b9a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:42.170365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6efb12e-295e-4f90-bc32-395949e0b9a0 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:42.182097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1029/2059] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:31:42.618501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447d51e4-26c0-4ef1-ac85-c32b375eb19d None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:42.620862 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-447d51e4-26c0-4ef1-ac85-c32b375eb19d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:42.639474 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447d51e4-26c0-4ef1-ac85-c32b375eb19d None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:42.647522 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:42.648297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:42.648517 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:42.648776 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c admin admin] PATH_INFO: `/v3/projects/9cb8da2b8315493387c4c340b31fee37` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:42.650117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c admin admin] RBAC: Authorizing `identity:delete_project(project_id=9cb8da2b8315493387c4c340b31fee37)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:42.653138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:42.765395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9cb8da2b8315493387c4c340b31fee37', 'request_id': 'req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:42.771846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:42.782613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9cb8da2b8315493387c4c340b31fee37', 'request_id': 'req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:42.796670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9cb8da2b8315493387c4c340b31fee37', 'request_id': 'req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:42.819412 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c admin admin] The token cache is being invalidate because project 9cb8da2b8315493387c4c340b31fee37 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:31:42.820087 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:42.821471 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:42.825207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1031/2060] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:31:42 2022] DELETE /identity/v3/projects/9cb8da2b8315493387c4c340b31fee37 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:42.830058 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6efb12e-295e-4f90-bc32-395949e0b9a0 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:42.844704 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6efb12e-295e-4f90-bc32-395949e0b9a0 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:42.924103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af66180f-3f1a-4bc8-a4a1-90ea0b2372df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:42.924755 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af66180f-3f1a-4bc8-a4a1-90ea0b2372df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:42.925082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af66180f-3f1a-4bc8-a4a1-90ea0b2372df admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:42.925285 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af66180f-3f1a-4bc8-a4a1-90ea0b2372df admin admin] PATH_INFO: `/v3/users/74117a5146774d17b355fa590c0f3e44` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:42.926467 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af66180f-3f1a-4bc8-a4a1-90ea0b2372df admin admin] RBAC: Authorizing `identity:delete_user(user_id=74117a5146774d17b355fa590c0f3e44)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:42.928574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af66180f-3f1a-4bc8-a4a1-90ea0b2372df admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:42.952395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:42.954041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:42.962181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:43.000297 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-af66180f-3f1a-4bc8-a4a1-90ea0b2372df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '74117a5146774d17b355fa590c0f3e44'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:43.010385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-af66180f-3f1a-4bc8-a4a1-90ea0b2372df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '74117a5146774d17b355fa590c0f3e44'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:43.020878 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-af66180f-3f1a-4bc8-a4a1-90ea0b2372df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '74117a5146774d17b355fa590c0f3e44'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:43.035235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-af66180f-3f1a-4bc8-a4a1-90ea0b2372df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '74117a5146774d17b355fa590c0f3e44'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:43.036211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-af66180f-3f1a-4bc8-a4a1-90ea0b2372df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '74117a5146774d17b355fa590c0f3e44'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:43.040452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c5bfbd-95f4-4d4f-9fda-2aba4f852e7c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:43.044181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1030/2061] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:31:42 2022] DELETE /identity/v3/users/74117a5146774d17b355fa590c0f3e44 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:43.044917 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e7e652-cf08-48f4-a4c0-8af25f222166 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:43.045730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e7e652-cf08-48f4-a4c0-8af25f222166 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:43.046159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e7e652-cf08-48f4-a4c0-8af25f222166 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:43.046574 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e7e652-cf08-48f4-a4c0-8af25f222166 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:43.049174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-00e7e652-cf08-48f4-a4c0-8af25f222166 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:43.140826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e7e652-cf08-48f4-a4c0-8af25f222166 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:43.142995 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e7e652-cf08-48f4-a4c0-8af25f222166 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:43.160942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1032/2062] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:42 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:31:43.457189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af66180f-3f1a-4bc8-a4a1-90ea0b2372df admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:43.459072 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-af66180f-3f1a-4bc8-a4a1-90ea0b2372df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:43.571520 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af66180f-3f1a-4bc8-a4a1-90ea0b2372df admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:43.580159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c364cf45-ff72-4638-a6a7-70f1de0bc64c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:43.581295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c364cf45-ff72-4638-a6a7-70f1de0bc64c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:43.582228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c364cf45-ff72-4638-a6a7-70f1de0bc64c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:43.582228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c364cf45-ff72-4638-a6a7-70f1de0bc64c admin admin] PATH_INFO: `/v3/projects/ac05002c2603474394372c9e94b5e5c0` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:43.584450 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c364cf45-ff72-4638-a6a7-70f1de0bc64c admin admin] RBAC: Authorizing `identity:delete_project(project_id=ac05002c2603474394372c9e94b5e5c0)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:43.588558 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c364cf45-ff72-4638-a6a7-70f1de0bc64c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:43.684613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c364cf45-ff72-4638-a6a7-70f1de0bc64c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ac05002c2603474394372c9e94b5e5c0', 'request_id': 'req-c364cf45-ff72-4638-a6a7-70f1de0bc64c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:43.697398 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c364cf45-ff72-4638-a6a7-70f1de0bc64c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ac05002c2603474394372c9e94b5e5c0', 'request_id': 'req-c364cf45-ff72-4638-a6a7-70f1de0bc64c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:43.717443 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c364cf45-ff72-4638-a6a7-70f1de0bc64c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ac05002c2603474394372c9e94b5e5c0', 'request_id': 'req-c364cf45-ff72-4638-a6a7-70f1de0bc64c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:43.750602 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c364cf45-ff72-4638-a6a7-70f1de0bc64c admin admin] The token cache is being invalidate because project ac05002c2603474394372c9e94b5e5c0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:31:43.751202 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c364cf45-ff72-4638-a6a7-70f1de0bc64c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:43.752955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c364cf45-ff72-4638-a6a7-70f1de0bc64c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:43.755599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1031/2063] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:31:43 2022] DELETE /identity/v3/projects/ac05002c2603474394372c9e94b5e5c0 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:31:43.760609 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e516d456-caa8-4a95-b1b7-46c3bc7317a8 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:43.761172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e516d456-caa8-4a95-b1b7-46c3bc7317a8 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:43.761172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e516d456-caa8-4a95-b1b7-46c3bc7317a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:43.811236 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-e516d456-caa8-4a95-b1b7-46c3bc7317a8 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:43.812745 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-e516d456-caa8-4a95-b1b7-46c3bc7317a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:43.904803 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1033/2064] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:43.920579 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5768b568-2810-4428-871f-0bcac6a3d187 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:43.921793 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5768b568-2810-4428-871f-0bcac6a3d187 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:43.922533 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5768b568-2810-4428-871f-0bcac6a3d187 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:43.964737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-5768b568-2810-4428-871f-0bcac6a3d187 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:43.965982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5768b568-2810-4428-871f-0bcac6a3d187 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:43.989118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1032/2065] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:43.990473 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e516d456-caa8-4a95-b1b7-46c3bc7317a8 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:44.002240 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e516d456-caa8-4a95-b1b7-46c3bc7317a8 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:44.009807 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bb8893-319e-4e12-91e6-d99f8cc3a66a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:44.010754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bb8893-319e-4e12-91e6-d99f8cc3a66a admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:44.011045 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bb8893-319e-4e12-91e6-d99f8cc3a66a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:44.011215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bb8893-319e-4e12-91e6-d99f8cc3a66a admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:44.011941 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bb8893-319e-4e12-91e6-d99f8cc3a66a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:31:44.012369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bb8893-319e-4e12-91e6-d99f8cc3a66a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:44.014548 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bb8893-319e-4e12-91e6-d99f8cc3a66a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:44.028561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5768b568-2810-4428-871f-0bcac6a3d187 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:44.030244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5768b568-2810-4428-871f-0bcac6a3d187 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:44.042486 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1034/2066] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:31:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:31:44.043944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bb8893-319e-4e12-91e6-d99f8cc3a66a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:44.066652 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bb8893-319e-4e12-91e6-d99f8cc3a66a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:44.069048 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:44.074595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98095a90-b665-4267-9e14-5e85f72ab5ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:44.075586 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98095a90-b665-4267-9e14-5e85f72ab5ba admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:44.075942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98095a90-b665-4267-9e14-5e85f72ab5ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:44.076296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98095a90-b665-4267-9e14-5e85f72ab5ba admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:44.077409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98095a90-b665-4267-9e14-5e85f72ab5ba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:44.084884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98095a90-b665-4267-9e14-5e85f72ab5ba admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:44.123875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5768b568-2810-4428-871f-0bcac6a3d187 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:44.127450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=95,keystone:DELETE=3,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:44.128098 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7312fa1-e177-4ca4-9820-250d36143ac5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:44.128932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7312fa1-e177-4ca4-9820-250d36143ac5 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:44.129306 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7312fa1-e177-4ca4-9820-250d36143ac5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:44.129625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7312fa1-e177-4ca4-9820-250d36143ac5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:44.131704 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f7312fa1-e177-4ca4-9820-250d36143ac5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:44.136690 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1035/2067] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:31:44 2022] POST /identity/v3/projects => generated 377 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:44.143140 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98095a90-b665-4267-9e14-5e85f72ab5ba admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:44.217516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7312fa1-e177-4ca4-9820-250d36143ac5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:44.219045 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7312fa1-e177-4ca4-9820-250d36143ac5 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:44.230530 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98095a90-b665-4267-9e14-5e85f72ab5ba admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:44.232417 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1033/2068] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:44 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:31:44.236058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1348f5-f363-420a-8af8-d29c081dccb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:44.236718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1348f5-f363-420a-8af8-d29c081dccb5 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:44.237001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1348f5-f363-420a-8af8-d29c081dccb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:44.237882 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1348f5-f363-420a-8af8-d29c081dccb5 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:44.238872 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1348f5-f363-420a-8af8-d29c081dccb5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:44.240843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1348f5-f363-420a-8af8-d29c081dccb5 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:44.267320 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1036/2069] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:31:44 2022] POST /identity/v3/users => generated 289 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:44.272425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7312fa1-e177-4ca4-9820-250d36143ac5 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:44.285114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7312fa1-e177-4ca4-9820-250d36143ac5 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:44.291452 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147f0aaa-d4d8-40c9-8d65-17bb8ca5c129 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:44.292080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147f0aaa-d4d8-40c9-8d65-17bb8ca5c129 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:44.292394 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147f0aaa-d4d8-40c9-8d65-17bb8ca5c129 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:44.292671 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147f0aaa-d4d8-40c9-8d65-17bb8ca5c129 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:44.293323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147f0aaa-d4d8-40c9-8d65-17bb8ca5c129 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:44.294955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147f0aaa-d4d8-40c9-8d65-17bb8ca5c129 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:44.312237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1034/2070] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:44 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:44.313556 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1348f5-f363-420a-8af8-d29c081dccb5 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:44.324545 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1348f5-f363-420a-8af8-d29c081dccb5 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:44.329428 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8154b41-3703-43c9-96d6-e3fc342c4939 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:44.330067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8154b41-3703-43c9-96d6-e3fc342c4939 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:44.330342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8154b41-3703-43c9-96d6-e3fc342c4939 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:44.330609 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8154b41-3703-43c9-96d6-e3fc342c4939 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:44.331828 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8154b41-3703-43c9-96d6-e3fc342c4939 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:44.333483 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8154b41-3703-43c9-96d6-e3fc342c4939 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:44.348224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1037/2071] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:44.351821 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147f0aaa-d4d8-40c9-8d65-17bb8ca5c129 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:44.366520 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147f0aaa-d4d8-40c9-8d65-17bb8ca5c129 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:44.372018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47368f3-07a4-4605-9df1-36decef37c5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:44.372751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47368f3-07a4-4605-9df1-36decef37c5c admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:44.373186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47368f3-07a4-4605-9df1-36decef37c5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:44.373462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47368f3-07a4-4605-9df1-36decef37c5c admin admin] PATH_INFO: `/v3/projects/bfe2ea258beb497992d54059260a05c3/users/046d03ada4c34ae18b14f2e7b95dbf81/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:44.391926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47368f3-07a4-4605-9df1-36decef37c5c admin admin] RBAC: Authorizing `identity:create_grant(project_id=bfe2ea258beb497992d54059260a05c3, user_id=046d03ada4c34ae18b14f2e7b95dbf81, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:44.395753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47368f3-07a4-4605-9df1-36decef37c5c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:44.414536 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1035/2072] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:44 2022] PUT /identity/v3/projects/bfe2ea258beb497992d54059260a05c3/users/046d03ada4c34ae18b14f2e7b95dbf81/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:44.428299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8002fd34-be4a-4532-870b-b89d4b423127 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:44.428618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8002fd34-be4a-4532-870b-b89d4b423127 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:44.428899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8002fd34-be4a-4532-870b-b89d4b423127 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:44.465649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-8002fd34-be4a-4532-870b-b89d4b423127 None None] MFA Rules not processed for user `046d03ada4c34ae18b14f2e7b95dbf81`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:44.465649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8002fd34-be4a-4532-870b-b89d4b423127 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:44.547546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1038/2073] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:44 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:44.552699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47368f3-07a4-4605-9df1-36decef37c5c admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:44.626086 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47368f3-07a4-4605-9df1-36decef37c5c admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:44.631804 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4619e8f5-f311-4e83-87aa-e1225d631759 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:44.632444 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4619e8f5-f311-4e83-87aa-e1225d631759 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:44.632728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4619e8f5-f311-4e83-87aa-e1225d631759 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:44.633613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4619e8f5-f311-4e83-87aa-e1225d631759 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:44.634262 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4619e8f5-f311-4e83-87aa-e1225d631759 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:44.636390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4619e8f5-f311-4e83-87aa-e1225d631759 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:44.671460 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1036/2074] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:31:44 2022] POST /identity/v3/projects => generated 375 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:44.672678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8002fd34-be4a-4532-870b-b89d4b423127 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:44.686046 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8002fd34-be4a-4532-870b-b89d4b423127 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:44.750918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bcea14-845e-4dc2-aacd-bc3526b7ebfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:44.751443 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bcea14-845e-4dc2-aacd-bc3526b7ebfd admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:44.751722 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bcea14-845e-4dc2-aacd-bc3526b7ebfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:44.752013 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bcea14-845e-4dc2-aacd-bc3526b7ebfd admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:44.752885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bcea14-845e-4dc2-aacd-bc3526b7ebfd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:44.756092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bcea14-845e-4dc2-aacd-bc3526b7ebfd admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:44.774202 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:44.781276 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1039/2075] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:31:44 2022] POST /identity/v3/users => generated 294 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:31:44.786501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4619e8f5-f311-4e83-87aa-e1225d631759 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:44.798568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4619e8f5-f311-4e83-87aa-e1225d631759 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:44.807244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1c3084-04bb-4b1a-ac68-7bcbb6781b99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:44.808587 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1c3084-04bb-4b1a-ac68-7bcbb6781b99 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:44.808947 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1c3084-04bb-4b1a-ac68-7bcbb6781b99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:44.809381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1c3084-04bb-4b1a-ac68-7bcbb6781b99 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:44.810108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1c3084-04bb-4b1a-ac68-7bcbb6781b99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:44.811782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1c3084-04bb-4b1a-ac68-7bcbb6781b99 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:44.832907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1037/2076] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:44 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:44.837579 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bcea14-845e-4dc2-aacd-bc3526b7ebfd admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:44.882157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bcea14-845e-4dc2-aacd-bc3526b7ebfd admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:44.888187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18ea52e-21e1-45a1-a1ce-0136c32be6b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:44.888959 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18ea52e-21e1-45a1-a1ce-0136c32be6b2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:44.889335 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18ea52e-21e1-45a1-a1ce-0136c32be6b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:44.889621 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18ea52e-21e1-45a1-a1ce-0136c32be6b2 admin admin] PATH_INFO: `/v3/projects/929475f7f7984fda808ef3aac39bbf3f/users/798c86ea7ce847e6b2dda58ba4af88e2/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:44.919886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18ea52e-21e1-45a1-a1ce-0136c32be6b2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=929475f7f7984fda808ef3aac39bbf3f, user_id=798c86ea7ce847e6b2dda58ba4af88e2, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:44.923286 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18ea52e-21e1-45a1-a1ce-0136c32be6b2 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:44.956854 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1040/2077] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:44 2022] PUT /identity/v3/projects/929475f7f7984fda808ef3aac39bbf3f/users/798c86ea7ce847e6b2dda58ba4af88e2/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:44.961106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1c3084-04bb-4b1a-ac68-7bcbb6781b99 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:44.975439 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1c3084-04bb-4b1a-ac68-7bcbb6781b99 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:44.998121 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:45.070951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1079e806-6b90-4e2c-9445-7b614fb62ca1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:45.071657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1079e806-6b90-4e2c-9445-7b614fb62ca1 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:45.071941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1079e806-6b90-4e2c-9445-7b614fb62ca1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:45.072237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1079e806-6b90-4e2c-9445-7b614fb62ca1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:45.072906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1079e806-6b90-4e2c-9445-7b614fb62ca1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:45.074503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1079e806-6b90-4e2c-9445-7b614fb62ca1 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:45.085983 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1038/2078] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:44 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:45.090170 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18ea52e-21e1-45a1-a1ce-0136c32be6b2 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:45.101481 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18ea52e-21e1-45a1-a1ce-0136c32be6b2 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:45.106255 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d68b76-85d4-4884-8e93-47494a56a472 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:45.107023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d68b76-85d4-4884-8e93-47494a56a472 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:45.107214 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d68b76-85d4-4884-8e93-47494a56a472 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:45.107481 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d68b76-85d4-4884-8e93-47494a56a472 admin admin] PATH_INFO: `/v3/projects/929475f7f7984fda808ef3aac39bbf3f/users/798c86ea7ce847e6b2dda58ba4af88e2/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:45.109470 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d68b76-85d4-4884-8e93-47494a56a472 admin admin] RBAC: Authorizing `identity:create_grant(project_id=929475f7f7984fda808ef3aac39bbf3f, user_id=798c86ea7ce847e6b2dda58ba4af88e2, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:45.112181 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d68b76-85d4-4884-8e93-47494a56a472 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:45.129010 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1041/2079] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:45 2022] PUT /identity/v3/projects/929475f7f7984fda808ef3aac39bbf3f/users/798c86ea7ce847e6b2dda58ba4af88e2/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:45.142823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041caf85-7686-4838-a03c-aa211a070af6 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:45.143101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041caf85-7686-4838-a03c-aa211a070af6 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:45.143511 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041caf85-7686-4838-a03c-aa211a070af6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:45.184791 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-041caf85-7686-4838-a03c-aa211a070af6 None None] MFA Rules not processed for user `798c86ea7ce847e6b2dda58ba4af88e2`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:45.186038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-041caf85-7686-4838-a03c-aa211a070af6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:45.278550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1039/2080] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:45 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:45.298334 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d68b76-85d4-4884-8e93-47494a56a472 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:45.300862 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-28d68b76-85d4-4884-8e93-47494a56a472 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:45.363528 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d68b76-85d4-4884-8e93-47494a56a472 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:45.367635 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56053e5-d5cb-49a5-a72e-501d5e74fab2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:45.368274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56053e5-d5cb-49a5-a72e-501d5e74fab2 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:45.368588 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56053e5-d5cb-49a5-a72e-501d5e74fab2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:45.368895 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56053e5-d5cb-49a5-a72e-501d5e74fab2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:45.443419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56053e5-d5cb-49a5-a72e-501d5e74fab2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:45.446393 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56053e5-d5cb-49a5-a72e-501d5e74fab2 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:45.466641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1042/2081] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:45 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:31:48.426505 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041caf85-7686-4838-a03c-aa211a070af6 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:48.428918 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-041caf85-7686-4838-a03c-aa211a070af6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:48.534493 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041caf85-7686-4838-a03c-aa211a070af6 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:48.539617 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947da162-5a13-4287-9437-18eeee17f361 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:48.540267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947da162-5a13-4287-9437-18eeee17f361 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:48.540580 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947da162-5a13-4287-9437-18eeee17f361 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:48.540851 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947da162-5a13-4287-9437-18eeee17f361 admin admin] PATH_INFO: `/v3/users/eec0c3bc09ab4d8c86528bb21b6ba9c2` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:48.542016 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947da162-5a13-4287-9437-18eeee17f361 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eec0c3bc09ab4d8c86528bb21b6ba9c2)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:48.543972 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947da162-5a13-4287-9437-18eeee17f361 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:48.594857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:48.610341 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-947da162-5a13-4287-9437-18eeee17f361 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eec0c3bc09ab4d8c86528bb21b6ba9c2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:48.622493 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-947da162-5a13-4287-9437-18eeee17f361 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eec0c3bc09ab4d8c86528bb21b6ba9c2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:48.634075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-947da162-5a13-4287-9437-18eeee17f361 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eec0c3bc09ab4d8c86528bb21b6ba9c2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:48.660941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-947da162-5a13-4287-9437-18eeee17f361 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eec0c3bc09ab4d8c86528bb21b6ba9c2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:48.662077 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-947da162-5a13-4287-9437-18eeee17f361 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eec0c3bc09ab4d8c86528bb21b6ba9c2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:48.673168 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1040/2082] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:31:48 2022] DELETE /identity/v3/users/eec0c3bc09ab4d8c86528bb21b6ba9c2 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:48.681844 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e940033-78e4-4397-878e-88409dd87cbf None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:48.682429 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e940033-78e4-4397-878e-88409dd87cbf None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:48.682837 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e940033-78e4-4397-878e-88409dd87cbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:48.712842 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-0e940033-78e4-4397-878e-88409dd87cbf None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:48.714563 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0e940033-78e4-4397-878e-88409dd87cbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:48.837311 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1043/2083] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:48.849463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947da162-5a13-4287-9437-18eeee17f361 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:48.855135 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-947da162-5a13-4287-9437-18eeee17f361 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:48.925452 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947da162-5a13-4287-9437-18eeee17f361 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:48.929263 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a3a8a5-7cbb-4801-8095-7e3486d2e406 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:48.929909 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a3a8a5-7cbb-4801-8095-7e3486d2e406 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:48.930185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a3a8a5-7cbb-4801-8095-7e3486d2e406 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:48.930455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a3a8a5-7cbb-4801-8095-7e3486d2e406 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:48.941911 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a3a8a5-7cbb-4801-8095-7e3486d2e406 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:48.943606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a3a8a5-7cbb-4801-8095-7e3486d2e406 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:48.954350 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1041/2084] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:31:49.272107 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e940033-78e4-4397-878e-88409dd87cbf None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:49.273622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0e940033-78e4-4397-878e-88409dd87cbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:49.288468 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e940033-78e4-4397-878e-88409dd87cbf None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:49.295379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:49.295874 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:49.296176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:49.296542 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595 admin admin] PATH_INFO: `/v3/projects/280c40e0abb743e0bbeb98cd37b50065` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:49.300137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595 admin admin] RBAC: Authorizing `identity:delete_project(project_id=280c40e0abb743e0bbeb98cd37b50065)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:49.300395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:49.329383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:49.376611 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '280c40e0abb743e0bbeb98cd37b50065', 'request_id': 'req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:49.390006 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '280c40e0abb743e0bbeb98cd37b50065', 'request_id': 'req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:49.395446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '280c40e0abb743e0bbeb98cd37b50065', 'request_id': 'req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:49.419171 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595 admin admin] The token cache is being invalidate because project 280c40e0abb743e0bbeb98cd37b50065 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:31:49.419461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:49.420100 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:49.421608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1044/2085] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:31:49 2022] DELETE /identity/v3/projects/280c40e0abb743e0bbeb98cd37b50065 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:49.426476 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a3a8a5-7cbb-4801-8095-7e3486d2e406 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:49.442180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a3a8a5-7cbb-4801-8095-7e3486d2e406 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:49.509987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e2571b-e75a-4131-a282-4ad9b5f10bbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:49.510631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e2571b-e75a-4131-a282-4ad9b5f10bbf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:49.510905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e2571b-e75a-4131-a282-4ad9b5f10bbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:49.511175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e2571b-e75a-4131-a282-4ad9b5f10bbf admin admin] PATH_INFO: `/v3/users/07e152b1680a4121b8e9d4cb59f6560b` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:49.512396 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e2571b-e75a-4131-a282-4ad9b5f10bbf admin admin] RBAC: Authorizing `identity:delete_user(user_id=07e152b1680a4121b8e9d4cb59f6560b)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:49.514371 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e2571b-e75a-4131-a282-4ad9b5f10bbf admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:49.573384 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-60e2571b-e75a-4131-a282-4ad9b5f10bbf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '07e152b1680a4121b8e9d4cb59f6560b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:49.582329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-60e2571b-e75a-4131-a282-4ad9b5f10bbf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '07e152b1680a4121b8e9d4cb59f6560b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:49.587627 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:49.592466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-60e2571b-e75a-4131-a282-4ad9b5f10bbf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '07e152b1680a4121b8e9d4cb59f6560b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:49.606359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-60e2571b-e75a-4131-a282-4ad9b5f10bbf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '07e152b1680a4121b8e9d4cb59f6560b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:49.607065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-60e2571b-e75a-4131-a282-4ad9b5f10bbf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '07e152b1680a4121b8e9d4cb59f6560b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:49.613191 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1042/2086] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:31:49 2022] DELETE /identity/v3/users/07e152b1680a4121b8e9d4cb59f6560b => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:49.930255 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:49.931727 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:49.973791 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e8b9ee-45ac-45b0-a7e8-1622ee3f7595 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:50.053991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1a2664-ac91-4181-843a-c12d7d9b4186 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:50.054652 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1a2664-ac91-4181-843a-c12d7d9b4186 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:50.054932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1a2664-ac91-4181-843a-c12d7d9b4186 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:50.055218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1a2664-ac91-4181-843a-c12d7d9b4186 admin admin] PATH_INFO: `/v3/projects/644f0263851d4b90917847e9c3c0b8cc` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:50.058214 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1a2664-ac91-4181-843a-c12d7d9b4186 admin admin] RBAC: Authorizing `identity:delete_project(project_id=644f0263851d4b90917847e9c3c0b8cc)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:50.060461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1a2664-ac91-4181-843a-c12d7d9b4186 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:50.121096 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8c1a2664-ac91-4181-843a-c12d7d9b4186 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '644f0263851d4b90917847e9c3c0b8cc', 'request_id': 'req-8c1a2664-ac91-4181-843a-c12d7d9b4186'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:50.127926 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8c1a2664-ac91-4181-843a-c12d7d9b4186 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '644f0263851d4b90917847e9c3c0b8cc', 'request_id': 'req-8c1a2664-ac91-4181-843a-c12d7d9b4186'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:50.135072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8c1a2664-ac91-4181-843a-c12d7d9b4186 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '644f0263851d4b90917847e9c3c0b8cc', 'request_id': 'req-8c1a2664-ac91-4181-843a-c12d7d9b4186'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:50.150058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8c1a2664-ac91-4181-843a-c12d7d9b4186 admin admin] The token cache is being invalidate because project 644f0263851d4b90917847e9c3c0b8cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:31:50.160516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8c1a2664-ac91-4181-843a-c12d7d9b4186 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:50.160516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8c1a2664-ac91-4181-843a-c12d7d9b4186 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:50.160516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1045/2087] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:31:49 2022] DELETE /identity/v3/projects/644f0263851d4b90917847e9c3c0b8cc => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:50.160983 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e2571b-e75a-4131-a282-4ad9b5f10bbf admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:50.160983 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-60e2571b-e75a-4131-a282-4ad9b5f10bbf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:50.174099 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e2571b-e75a-4131-a282-4ad9b5f10bbf admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:50.261018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efae64a-bbd1-48ec-820a-dcd96e8356b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:50.262013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efae64a-bbd1-48ec-820a-dcd96e8356b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:50.262493 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efae64a-bbd1-48ec-820a-dcd96e8356b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:50.262941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efae64a-bbd1-48ec-820a-dcd96e8356b0 admin admin] PATH_INFO: `/v3/users/d37a20c847eb4d599b8ddb4822373c13` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:50.264813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efae64a-bbd1-48ec-820a-dcd96e8356b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d37a20c847eb4d599b8ddb4822373c13)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:50.269255 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efae64a-bbd1-48ec-820a-dcd96e8356b0 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:50.329404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8efae64a-bbd1-48ec-820a-dcd96e8356b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd37a20c847eb4d599b8ddb4822373c13'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:50.337238 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8efae64a-bbd1-48ec-820a-dcd96e8356b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd37a20c847eb4d599b8ddb4822373c13'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:50.346090 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8efae64a-bbd1-48ec-820a-dcd96e8356b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd37a20c847eb4d599b8ddb4822373c13'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:50.360366 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8efae64a-bbd1-48ec-820a-dcd96e8356b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd37a20c847eb4d599b8ddb4822373c13'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:50.361177 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8efae64a-bbd1-48ec-820a-dcd96e8356b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd37a20c847eb4d599b8ddb4822373c13'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:50.368720 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1043/2088] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:31:50 2022] DELETE /identity/v3/users/d37a20c847eb4d599b8ddb4822373c13 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:31:50.681206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1a2664-ac91-4181-843a-c12d7d9b4186 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:50.682651 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8c1a2664-ac91-4181-843a-c12d7d9b4186 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:50.695726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1a2664-ac91-4181-843a-c12d7d9b4186 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:50.724486 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:50.760127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b72d377-404f-4cde-9124-2fd073266679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:50.760681 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b72d377-404f-4cde-9124-2fd073266679 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:50.760963 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b72d377-404f-4cde-9124-2fd073266679 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:50.761238 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b72d377-404f-4cde-9124-2fd073266679 admin admin] PATH_INFO: `/v3/projects/b4b5cb0a3b704fbd9ae1bfea14b55a6e` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:50.762380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b72d377-404f-4cde-9124-2fd073266679 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b4b5cb0a3b704fbd9ae1bfea14b55a6e)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:50.764287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b72d377-404f-4cde-9124-2fd073266679 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:50.817071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0b72d377-404f-4cde-9124-2fd073266679 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b4b5cb0a3b704fbd9ae1bfea14b55a6e', 'request_id': 'req-0b72d377-404f-4cde-9124-2fd073266679'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:50.824154 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0b72d377-404f-4cde-9124-2fd073266679 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b4b5cb0a3b704fbd9ae1bfea14b55a6e', 'request_id': 'req-0b72d377-404f-4cde-9124-2fd073266679'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:50.831784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0b72d377-404f-4cde-9124-2fd073266679 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b4b5cb0a3b704fbd9ae1bfea14b55a6e', 'request_id': 'req-0b72d377-404f-4cde-9124-2fd073266679'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:50.846150 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0b72d377-404f-4cde-9124-2fd073266679 admin admin] The token cache is being invalidate because project b4b5cb0a3b704fbd9ae1bfea14b55a6e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:31:50.850267 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0b72d377-404f-4cde-9124-2fd073266679 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:50.850267 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0b72d377-404f-4cde-9124-2fd073266679 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:50.850267 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1046/2089] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:31:50 2022] DELETE /identity/v3/projects/b4b5cb0a3b704fbd9ae1bfea14b55a6e => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:31:50.855162 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3738832-4f4c-4c63-bc13-1121869ec7a1 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:50.855932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3738832-4f4c-4c63-bc13-1121869ec7a1 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:50.856118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3738832-4f4c-4c63-bc13-1121869ec7a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:50.876335 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-c3738832-4f4c-4c63-bc13-1121869ec7a1 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:50.878060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c3738832-4f4c-4c63-bc13-1121869ec7a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:50.930500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:50.944749 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1044/2090] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:50.952490 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b018592-b489-4cc1-bf6b-410106d09afb None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:50.952983 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b018592-b489-4cc1-bf6b-410106d09afb None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:50.953078 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b018592-b489-4cc1-bf6b-410106d09afb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:50.974269 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-1b018592-b489-4cc1-bf6b-410106d09afb None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:50.975490 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-1b018592-b489-4cc1-bf6b-410106d09afb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:50.983499 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1047/2091] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:50.988077 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3738832-4f4c-4c63-bc13-1121869ec7a1 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:50.998070 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3738832-4f4c-4c63-bc13-1121869ec7a1 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:51.003415 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8fb0e2-9ea7-4368-9f42-173505f818b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:51.004358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8fb0e2-9ea7-4368-9f42-173505f818b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:51.004660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8fb0e2-9ea7-4368-9f42-173505f818b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:51.005099 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8fb0e2-9ea7-4368-9f42-173505f818b7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:51.005760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8fb0e2-9ea7-4368-9f42-173505f818b7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:31:51.006062 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8fb0e2-9ea7-4368-9f42-173505f818b7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:51.007665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8fb0e2-9ea7-4368-9f42-173505f818b7 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:51.021691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1045/2092] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:31:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:31:51.025391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b018592-b489-4cc1-bf6b-410106d09afb None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:51.035472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b018592-b489-4cc1-bf6b-410106d09afb None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:51.040853 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e522218e-9484-4c38-b5d1-16e1d222299b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:51.041469 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e522218e-9484-4c38-b5d1-16e1d222299b admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:51.041749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e522218e-9484-4c38-b5d1-16e1d222299b admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:51.042020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e522218e-9484-4c38-b5d1-16e1d222299b admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:51.042674 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e522218e-9484-4c38-b5d1-16e1d222299b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:51.044576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e522218e-9484-4c38-b5d1-16e1d222299b admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:51.076323 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1048/2093] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:31:51 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:51.080560 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8fb0e2-9ea7-4368-9f42-173505f818b7 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:51.092760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8fb0e2-9ea7-4368-9f42-173505f818b7 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:51.181514 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538b3376-9809-44da-923d-0a70abaceb9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:51.182647 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538b3376-9809-44da-923d-0a70abaceb9c admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:51.183236 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538b3376-9809-44da-923d-0a70abaceb9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:51.183846 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538b3376-9809-44da-923d-0a70abaceb9c admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:51.185214 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538b3376-9809-44da-923d-0a70abaceb9c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:51.187324 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538b3376-9809-44da-923d-0a70abaceb9c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:51.210135 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1046/2094] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:31:51 2022] POST /identity/v3/users => generated 293 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:31:51.213919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e522218e-9484-4c38-b5d1-16e1d222299b admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:51.225990 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e522218e-9484-4c38-b5d1-16e1d222299b admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:51.232029 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c20896-fe9b-4a7d-bdf3-80792baf5bee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:51.232685 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c20896-fe9b-4a7d-bdf3-80792baf5bee admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:51.232964 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c20896-fe9b-4a7d-bdf3-80792baf5bee admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:51.233260 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c20896-fe9b-4a7d-bdf3-80792baf5bee admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:51.233894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c20896-fe9b-4a7d-bdf3-80792baf5bee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:51.235498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c20896-fe9b-4a7d-bdf3-80792baf5bee admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:51.247093 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1049/2095] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:51 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:51.253427 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538b3376-9809-44da-923d-0a70abaceb9c admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:51.269774 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538b3376-9809-44da-923d-0a70abaceb9c admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:51.276646 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb51481-0e2e-4939-b739-5295a0a1ee08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:51.277524 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb51481-0e2e-4939-b739-5295a0a1ee08 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:51.278013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb51481-0e2e-4939-b739-5295a0a1ee08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:51.278438 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb51481-0e2e-4939-b739-5295a0a1ee08 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:51.279502 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb51481-0e2e-4939-b739-5295a0a1ee08 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:51.282147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb51481-0e2e-4939-b739-5295a0a1ee08 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:51.309159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1047/2096] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:51 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:51.313638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c20896-fe9b-4a7d-bdf3-80792baf5bee admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:51.326374 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c20896-fe9b-4a7d-bdf3-80792baf5bee admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:51.333465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f876f90-a2f5-422d-a58e-efafa4430f5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:51.334137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f876f90-a2f5-422d-a58e-efafa4430f5c admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:51.334798 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f876f90-a2f5-422d-a58e-efafa4430f5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:51.334798 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f876f90-a2f5-422d-a58e-efafa4430f5c admin admin] PATH_INFO: `/v3/projects/ffff627ea1ff4f7ea8b9da84b38760f0/users/b94a312e5e7344d5806478abae071d4e/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:51.352541 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f876f90-a2f5-422d-a58e-efafa4430f5c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ffff627ea1ff4f7ea8b9da84b38760f0, user_id=b94a312e5e7344d5806478abae071d4e, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:51.361475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f876f90-a2f5-422d-a58e-efafa4430f5c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:51.381312 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1050/2097] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:51 2022] PUT /identity/v3/projects/ffff627ea1ff4f7ea8b9da84b38760f0/users/b94a312e5e7344d5806478abae071d4e/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:31:51.391369 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7d3422-acf8-42fb-bbd7-856bfa69f827 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:51.391651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7d3422-acf8-42fb-bbd7-856bfa69f827 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:51.391924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7d3422-acf8-42fb-bbd7-856bfa69f827 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:51.428142 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-ff7d3422-acf8-42fb-bbd7-856bfa69f827 None None] MFA Rules not processed for user `b94a312e5e7344d5806478abae071d4e`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:51.429384 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ff7d3422-acf8-42fb-bbd7-856bfa69f827 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:51.539219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1048/2098] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:51 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:51.553892 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f876f90-a2f5-422d-a58e-efafa4430f5c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:51.555670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8f876f90-a2f5-422d-a58e-efafa4430f5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:51.613326 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f876f90-a2f5-422d-a58e-efafa4430f5c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:51.617090 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455a22ab-0b28-4ad2-9a74-03e567b662af service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:51.617674 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455a22ab-0b28-4ad2-9a74-03e567b662af service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:51.617952 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455a22ab-0b28-4ad2-9a74-03e567b662af service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:51.618223 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455a22ab-0b28-4ad2-9a74-03e567b662af service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:51.627057 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455a22ab-0b28-4ad2-9a74-03e567b662af service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:51.628680 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455a22ab-0b28-4ad2-9a74-03e567b662af service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:51.632230 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:51.639475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1051/2099] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:51 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:31:52.590452 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7d3422-acf8-42fb-bbd7-856bfa69f827 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:52.592245 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ff7d3422-acf8-42fb-bbd7-856bfa69f827 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:52.610393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7d3422-acf8-42fb-bbd7-856bfa69f827 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:52.614729 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866cd203-c113-4f92-916a-3b5ef78c26cd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:52.615419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866cd203-c113-4f92-916a-3b5ef78c26cd service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:52.615606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866cd203-c113-4f92-916a-3b5ef78c26cd service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:52.615885 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866cd203-c113-4f92-916a-3b5ef78c26cd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:52.617735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-866cd203-c113-4f92-916a-3b5ef78c26cd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:52.675606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:52.692372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866cd203-c113-4f92-916a-3b5ef78c26cd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:52.693817 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866cd203-c113-4f92-916a-3b5ef78c26cd service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:52.703545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1049/2100] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:52 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:31:57.411262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455a22ab-0b28-4ad2-9a74-03e567b662af service cinder] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:57.485039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455a22ab-0b28-4ad2-9a74-03e567b662af service cinder] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:57.490273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a1f22b-1926-4d93-bf12-324ec93d38da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:57.490971 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a1f22b-1926-4d93-bf12-324ec93d38da admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:57.491256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a1f22b-1926-4d93-bf12-324ec93d38da admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:57.491534 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a1f22b-1926-4d93-bf12-324ec93d38da admin admin] PATH_INFO: `/v3/users/b94a312e5e7344d5806478abae071d4e` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:57.492710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a1f22b-1926-4d93-bf12-324ec93d38da admin admin] RBAC: Authorizing `identity:delete_user(user_id=b94a312e5e7344d5806478abae071d4e)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:57.494650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a1f22b-1926-4d93-bf12-324ec93d38da admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:57.558881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-99a1f22b-1926-4d93-bf12-324ec93d38da admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b94a312e5e7344d5806478abae071d4e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:57.559850 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-99a1f22b-1926-4d93-bf12-324ec93d38da admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b94a312e5e7344d5806478abae071d4e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:57.561022 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-99a1f22b-1926-4d93-bf12-324ec93d38da admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b94a312e5e7344d5806478abae071d4e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:57.572773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-99a1f22b-1926-4d93-bf12-324ec93d38da admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b94a312e5e7344d5806478abae071d4e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:57.597830 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-99a1f22b-1926-4d93-bf12-324ec93d38da admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b94a312e5e7344d5806478abae071d4e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:57.616654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1052/2101] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:31:57 2022] DELETE /identity/v3/users/b94a312e5e7344d5806478abae071d4e => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:57.619044 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81319844-8202-418f-afc7-d09abdfcc6a8 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:57.619558 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81319844-8202-418f-afc7-d09abdfcc6a8 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:57.619962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81319844-8202-418f-afc7-d09abdfcc6a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:57.652507 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-81319844-8202-418f-afc7-d09abdfcc6a8 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:57.654657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-81319844-8202-418f-afc7-d09abdfcc6a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:57.739563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1050/2102] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:57.754072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a1f22b-1926-4d93-bf12-324ec93d38da admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:57.755616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-99a1f22b-1926-4d93-bf12-324ec93d38da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:57.828222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a1f22b-1926-4d93-bf12-324ec93d38da admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:57.833299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8192d8-e22f-4c80-8fcb-c777fde9881d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:57.834005 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8192d8-e22f-4c80-8fcb-c777fde9881d service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:57.834273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8192d8-e22f-4c80-8fcb-c777fde9881d service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:57.834550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8192d8-e22f-4c80-8fcb-c777fde9881d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:57.845621 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8192d8-e22f-4c80-8fcb-c777fde9881d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:57.847172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8192d8-e22f-4c80-8fcb-c777fde9881d service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:57.859318 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1053/2103] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:31:58.230116 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4180bf-a56a-4c23-aeb6-75d0d980ba2e None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:58.230381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4180bf-a56a-4c23-aeb6-75d0d980ba2e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:58.230662 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4180bf-a56a-4c23-aeb6-75d0d980ba2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:58.269804 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-8e4180bf-a56a-4c23-aeb6-75d0d980ba2e None None] MFA Rules not processed for user `c7028d2ccede4b44b567387bb83c4579`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:58.271482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-8e4180bf-a56a-4c23-aeb6-75d0d980ba2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:58.311909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8192d8-e22f-4c80-8fcb-c777fde9881d service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:58.313558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ea8192d8-e22f-4c80-8fcb-c777fde9881d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:58.388600 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8192d8-e22f-4c80-8fcb-c777fde9881d service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:58.401609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1051/2104] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:58 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:58.401923 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1aceffd-3782-463d-a32e-b3f290e00cc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:58.404099 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aceffd-3782-463d-a32e-b3f290e00cc4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:58.404934 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aceffd-3782-463d-a32e-b3f290e00cc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:58.405298 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aceffd-3782-463d-a32e-b3f290e00cc4 admin admin] PATH_INFO: `/v3/projects/ffff627ea1ff4f7ea8b9da84b38760f0` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:58.407109 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1aceffd-3782-463d-a32e-b3f290e00cc4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ffff627ea1ff4f7ea8b9da84b38760f0)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:58.415446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1aceffd-3782-463d-a32e-b3f290e00cc4 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:58.457470 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:58.520108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d1aceffd-3782-463d-a32e-b3f290e00cc4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ffff627ea1ff4f7ea8b9da84b38760f0', 'request_id': 'req-d1aceffd-3782-463d-a32e-b3f290e00cc4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:58.528007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d1aceffd-3782-463d-a32e-b3f290e00cc4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ffff627ea1ff4f7ea8b9da84b38760f0', 'request_id': 'req-d1aceffd-3782-463d-a32e-b3f290e00cc4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:58.540477 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d1aceffd-3782-463d-a32e-b3f290e00cc4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ffff627ea1ff4f7ea8b9da84b38760f0', 'request_id': 'req-d1aceffd-3782-463d-a32e-b3f290e00cc4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:58.558944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d1aceffd-3782-463d-a32e-b3f290e00cc4 admin admin] The token cache is being invalidate because project ffff627ea1ff4f7ea8b9da84b38760f0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:31:58.559278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d1aceffd-3782-463d-a32e-b3f290e00cc4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:58.560232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d1aceffd-3782-463d-a32e-b3f290e00cc4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:31:58.561912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1054/2105] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:31:58 2022] DELETE /identity/v3/projects/ffff627ea1ff4f7ea8b9da84b38760f0 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:58.569132 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c14d10-b04d-4743-a0d8-0e49579da05c None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:58.569778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c14d10-b04d-4743-a0d8-0e49579da05c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:58.570234 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c14d10-b04d-4743-a0d8-0e49579da05c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:58.597592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-88c14d10-b04d-4743-a0d8-0e49579da05c None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:58.598903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-88c14d10-b04d-4743-a0d8-0e49579da05c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:58.684507 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1052/2106] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:58.696092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bc9fba-55b8-46c0-a5bd-09c885bcbf6c None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:58.696794 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bc9fba-55b8-46c0-a5bd-09c885bcbf6c None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:58.697592 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bc9fba-55b8-46c0-a5bd-09c885bcbf6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:58.725805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-b9bc9fba-55b8-46c0-a5bd-09c885bcbf6c None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:58.727653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-b9bc9fba-55b8-46c0-a5bd-09c885bcbf6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:58.743873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1055/2107] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:31:58.749632 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c14d10-b04d-4743-a0d8-0e49579da05c None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:58.768694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c14d10-b04d-4743-a0d8-0e49579da05c None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:58.782899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c137355-84b8-4355-ad70-be8282246422 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:58.785636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c137355-84b8-4355-ad70-be8282246422 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:58.788295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c137355-84b8-4355-ad70-be8282246422 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:58.788722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c137355-84b8-4355-ad70-be8282246422 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:58.790542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c137355-84b8-4355-ad70-be8282246422 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:31:58.790542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c137355-84b8-4355-ad70-be8282246422 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:58.795681 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c137355-84b8-4355-ad70-be8282246422 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:58.824943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1053/2108] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:31:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:31:58.830654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bc9fba-55b8-46c0-a5bd-09c885bcbf6c None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:58.849048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bc9fba-55b8-46c0-a5bd-09c885bcbf6c None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:58.855579 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369a62b0-0713-4c3f-ba30-c59d5b3851e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:58.855871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369a62b0-0713-4c3f-ba30-c59d5b3851e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:58.856195 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369a62b0-0713-4c3f-ba30-c59d5b3851e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:58.856605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369a62b0-0713-4c3f-ba30-c59d5b3851e2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:58.857272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369a62b0-0713-4c3f-ba30-c59d5b3851e2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:58.859647 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369a62b0-0713-4c3f-ba30-c59d5b3851e2 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:58.909660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1056/2109] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:31:58 2022] POST /identity/v3/projects => generated 363 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:58.910075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c137355-84b8-4355-ad70-be8282246422 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:58.927249 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c137355-84b8-4355-ad70-be8282246422 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:58.948132 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:31:59.029593 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875ae6ea-c7aa-4e95-8521-8ae12520e39a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:59.030376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875ae6ea-c7aa-4e95-8521-8ae12520e39a admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:59.030669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875ae6ea-c7aa-4e95-8521-8ae12520e39a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:59.031169 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875ae6ea-c7aa-4e95-8521-8ae12520e39a admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:59.036571 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875ae6ea-c7aa-4e95-8521-8ae12520e39a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:59.040686 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875ae6ea-c7aa-4e95-8521-8ae12520e39a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:59.078936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1054/2110] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:31:58 2022] POST /identity/v3/users => generated 282 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:31:59.082055 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369a62b0-0713-4c3f-ba30-c59d5b3851e2 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:59.095751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369a62b0-0713-4c3f-ba30-c59d5b3851e2 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:59.102403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff6ebc8-9c96-48a0-9062-d96ed00e19fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:59.103288 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff6ebc8-9c96-48a0-9062-d96ed00e19fa admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:59.103778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff6ebc8-9c96-48a0-9062-d96ed00e19fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:59.104653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff6ebc8-9c96-48a0-9062-d96ed00e19fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:59.105549 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff6ebc8-9c96-48a0-9062-d96ed00e19fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:59.109878 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff6ebc8-9c96-48a0-9062-d96ed00e19fa admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:59.132000 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1057/2111] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:59 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:31:59.132639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875ae6ea-c7aa-4e95-8521-8ae12520e39a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:59.154160 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875ae6ea-c7aa-4e95-8521-8ae12520e39a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:59.160379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d51c3e-402d-44c7-b65b-7880c98d420a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:59.161011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d51c3e-402d-44c7-b65b-7880c98d420a admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:59.161287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d51c3e-402d-44c7-b65b-7880c98d420a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:59.161650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d51c3e-402d-44c7-b65b-7880c98d420a admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:59.162755 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d51c3e-402d-44c7-b65b-7880c98d420a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:59.164206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d51c3e-402d-44c7-b65b-7880c98d420a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:59.182804 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1055/2112] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:31:59 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:31:59.189644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff6ebc8-9c96-48a0-9062-d96ed00e19fa admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:59.205225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff6ebc8-9c96-48a0-9062-d96ed00e19fa admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:59.213994 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5b6ee2-e51c-42f1-af32-d14a0b8caa68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:59.214712 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5b6ee2-e51c-42f1-af32-d14a0b8caa68 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:59.215252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5b6ee2-e51c-42f1-af32-d14a0b8caa68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:59.215587 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5b6ee2-e51c-42f1-af32-d14a0b8caa68 admin admin] PATH_INFO: `/v3/projects/62dfe44d9fec40bdb293a1741670fc53/users/c22d02eea2cd4c9a8aacb54ca70c1ff3/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:59.239551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5b6ee2-e51c-42f1-af32-d14a0b8caa68 admin admin] RBAC: Authorizing `identity:create_grant(project_id=62dfe44d9fec40bdb293a1741670fc53, user_id=c22d02eea2cd4c9a8aacb54ca70c1ff3, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:59.242383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5b6ee2-e51c-42f1-af32-d14a0b8caa68 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:59.271339 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1058/2113] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:31:59 2022] PUT /identity/v3/projects/62dfe44d9fec40bdb293a1741670fc53/users/c22d02eea2cd4c9a8aacb54ca70c1ff3/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:31:59.282076 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86682801-4b37-4dcf-860f-82ef32ca1fd1 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:59.282670 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86682801-4b37-4dcf-860f-82ef32ca1fd1 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:59.282952 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86682801-4b37-4dcf-860f-82ef32ca1fd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:59.314040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-86682801-4b37-4dcf-860f-82ef32ca1fd1 None None] MFA Rules not processed for user `c22d02eea2cd4c9a8aacb54ca70c1ff3`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:31:59.316552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-86682801-4b37-4dcf-860f-82ef32ca1fd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:59.418198 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1056/2114] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:31:59 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:31:59.433307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5b6ee2-e51c-42f1-af32-d14a0b8caa68 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:31:59.435349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-6a5b6ee2-e51c-42f1-af32-d14a0b8caa68 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:31:59.521578 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5b6ee2-e51c-42f1-af32-d14a0b8caa68 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:31:59.524818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c98e2be-a1ed-4489-8b4c-e04a840230cc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:31:59.525675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c98e2be-a1ed-4489-8b4c-e04a840230cc service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:31:59.526044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c98e2be-a1ed-4489-8b4c-e04a840230cc service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:31:59.526400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c98e2be-a1ed-4489-8b4c-e04a840230cc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:31:59.537511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c98e2be-a1ed-4489-8b4c-e04a840230cc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:31:59.540330 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c98e2be-a1ed-4489-8b4c-e04a840230cc service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:31:59.559265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1059/2115] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:31:59 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:32:00.594717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6cf30c-61f4-42d1-9f21-1373f7e86b99 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:00.595230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6cf30c-61f4-42d1-9f21-1373f7e86b99 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:00.595541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6cf30c-61f4-42d1-9f21-1373f7e86b99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:00.616893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-1f6cf30c-61f4-42d1-9f21-1373f7e86b99 None None] MFA Rules not processed for user `54b3856eb61a487dab21d8ffa345701c`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:00.618296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1f6cf30c-61f4-42d1-9f21-1373f7e86b99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:00.700374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1057/2116] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:32:00 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:32:09.551718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=71,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:10.691962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=83,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:12.541721 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb88adf3-e3ba-4239-937b-72197614da75 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:12.542757 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb88adf3-e3ba-4239-937b-72197614da75 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:12.543046 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb88adf3-e3ba-4239-937b-72197614da75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:12.569466 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-fb88adf3-e3ba-4239-937b-72197614da75 None None] MFA Rules not processed for user `34afa4aac10c4a018ca05a0ba34c940a`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:12.570723 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-fb88adf3-e3ba-4239-937b-72197614da75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:12.636593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1060/2117] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Oct 4 14:32:12 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:32:12.714039 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6cf30c-61f4-42d1-9f21-1373f7e86b99 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:12.716751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1f6cf30c-61f4-42d1-9f21-1373f7e86b99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:12.865340 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6cf30c-61f4-42d1-9f21-1373f7e86b99 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:12.873985 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8e070b-7f53-4d17-a23f-cc1f537b6af8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5980e353164c4873bcdd2af847e0a1f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:12.874985 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8e070b-7f53-4d17-a23f-cc1f537b6af8 service swift] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:12.875474 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8e070b-7f53-4d17-a23f-cc1f537b6af8 service swift] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:12.876343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8e070b-7f53-4d17-a23f-cc1f537b6af8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:12.893628 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8e070b-7f53-4d17-a23f-cc1f537b6af8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:12.895793 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8e070b-7f53-4d17-a23f-cc1f537b6af8 service swift] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:12.908140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1058/2118] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Oct 4 14:32:12 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 04 14:32:22.629206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=25 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:22.906108 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=32 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:25.409764 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda440a1-b953-4e42-b78a-547e806e6efe None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:25.410919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda440a1-b953-4e42-b78a-547e806e6efe None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:25.411455 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda440a1-b953-4e42-b78a-547e806e6efe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:25.465480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-fda440a1-b953-4e42-b78a-547e806e6efe None None] MFA Rules not processed for user `34afa4aac10c4a018ca05a0ba34c940a`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:25.471855 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-fda440a1-b953-4e42-b78a-547e806e6efe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:25.507380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1061/2119] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Oct 4 14:32:25 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:32:25.537235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8e070b-7f53-4d17-a23f-cc1f537b6af8 service swift] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:25.552803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8e070b-7f53-4d17-a23f-cc1f537b6af8 service swift] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:25.570663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739b3358-aba7-4952-b353-6757742a5f70 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5980e353164c4873bcdd2af847e0a1f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:25.571443 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739b3358-aba7-4952-b353-6757742a5f70 service swift] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:25.572618 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739b3358-aba7-4952-b353-6757742a5f70 service swift] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:25.573011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739b3358-aba7-4952-b353-6757742a5f70 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:25.578438 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda440a1-b953-4e42-b78a-547e806e6efe None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:25.581060 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-fda440a1-b953-4e42-b78a-547e806e6efe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:25.618409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739b3358-aba7-4952-b353-6757742a5f70 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:25.640595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739b3358-aba7-4952-b353-6757742a5f70 service swift] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:25.680678 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1059/2120] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Oct 4 14:32:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 14:32:25.838747 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda440a1-b953-4e42-b78a-547e806e6efe None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:25.858280 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4c1daf-d671-4f01-8e81-9262b7a1f8b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:25.859526 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4c1daf-d671-4f01-8e81-9262b7a1f8b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:25.862590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4c1daf-d671-4f01-8e81-9262b7a1f8b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:25.864097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4c1daf-d671-4f01-8e81-9262b7a1f8b9 admin admin] PATH_INFO: `/v3/users/046d03ada4c34ae18b14f2e7b95dbf81` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:25.866844 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4c1daf-d671-4f01-8e81-9262b7a1f8b9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=046d03ada4c34ae18b14f2e7b95dbf81)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:25.874866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4c1daf-d671-4f01-8e81-9262b7a1f8b9 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:26.003440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5a4c1daf-d671-4f01-8e81-9262b7a1f8b9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '046d03ada4c34ae18b14f2e7b95dbf81'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:26.004549 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5a4c1daf-d671-4f01-8e81-9262b7a1f8b9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '046d03ada4c34ae18b14f2e7b95dbf81'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:26.005206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5a4c1daf-d671-4f01-8e81-9262b7a1f8b9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '046d03ada4c34ae18b14f2e7b95dbf81'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:26.026330 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5a4c1daf-d671-4f01-8e81-9262b7a1f8b9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '046d03ada4c34ae18b14f2e7b95dbf81'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:26.061751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5a4c1daf-d671-4f01-8e81-9262b7a1f8b9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '046d03ada4c34ae18b14f2e7b95dbf81'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:26.088264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1062/2121] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:32:25 2022] DELETE /identity/v3/users/046d03ada4c34ae18b14f2e7b95dbf81 => generated 0 bytes in 510 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:32:26.093093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ec417a-c1cc-42c5-b0dd-d49ff38bea68 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:26.093383 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ec417a-c1cc-42c5-b0dd-d49ff38bea68 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:26.093689 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ec417a-c1cc-42c5-b0dd-d49ff38bea68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:26.140457 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-c7ec417a-c1cc-42c5-b0dd-d49ff38bea68 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:26.142033 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c7ec417a-c1cc-42c5-b0dd-d49ff38bea68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:26.299928 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1060/2122] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:32:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:32:26.349875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4c1daf-d671-4f01-8e81-9262b7a1f8b9 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:26.352418 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-5a4c1daf-d671-4f01-8e81-9262b7a1f8b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:26.503302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4c1daf-d671-4f01-8e81-9262b7a1f8b9 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:26.518602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e03eb3-52a2-4ff3-9eea-211056047124 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:26.519353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e03eb3-52a2-4ff3-9eea-211056047124 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:26.519735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e03eb3-52a2-4ff3-9eea-211056047124 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:26.520114 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e03eb3-52a2-4ff3-9eea-211056047124 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:26.537304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e03eb3-52a2-4ff3-9eea-211056047124 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:26.539401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e03eb3-52a2-4ff3-9eea-211056047124 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:26.553266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1063/2123] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:32:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:32:28.426862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ec417a-c1cc-42c5-b0dd-d49ff38bea68 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:28.428793 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c7ec417a-c1cc-42c5-b0dd-d49ff38bea68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:28.460577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ec417a-c1cc-42c5-b0dd-d49ff38bea68 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:28.478942 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc697480-5710-4959-b7e9-6d0a3984cb76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:28.480254 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc697480-5710-4959-b7e9-6d0a3984cb76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:28.480851 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc697480-5710-4959-b7e9-6d0a3984cb76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:28.481246 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc697480-5710-4959-b7e9-6d0a3984cb76 admin admin] PATH_INFO: `/v3/projects/bfe2ea258beb497992d54059260a05c3` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:28.483202 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc697480-5710-4959-b7e9-6d0a3984cb76 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bfe2ea258beb497992d54059260a05c3)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:28.487399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc697480-5710-4959-b7e9-6d0a3984cb76 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:28.669022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-dc697480-5710-4959-b7e9-6d0a3984cb76 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bfe2ea258beb497992d54059260a05c3', 'request_id': 'req-dc697480-5710-4959-b7e9-6d0a3984cb76'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:28.681526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-dc697480-5710-4959-b7e9-6d0a3984cb76 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bfe2ea258beb497992d54059260a05c3', 'request_id': 'req-dc697480-5710-4959-b7e9-6d0a3984cb76'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:28.709215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-dc697480-5710-4959-b7e9-6d0a3984cb76 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bfe2ea258beb497992d54059260a05c3', 'request_id': 'req-dc697480-5710-4959-b7e9-6d0a3984cb76'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:28.751366 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-dc697480-5710-4959-b7e9-6d0a3984cb76 admin admin] The token cache is being invalidate because project bfe2ea258beb497992d54059260a05c3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:32:28.752280 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-dc697480-5710-4959-b7e9-6d0a3984cb76 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:28.753796 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-dc697480-5710-4959-b7e9-6d0a3984cb76 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:28.755842 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1061/2124] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:32:28 2022] DELETE /identity/v3/projects/bfe2ea258beb497992d54059260a05c3 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:32:28.760805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e03eb3-52a2-4ff3-9eea-211056047124 service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:28.781732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e03eb3-52a2-4ff3-9eea-211056047124 service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:28.830951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:28.980975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f858bf-bc1f-4972-b29b-dc1b9eddcffb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:28.983185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f858bf-bc1f-4972-b29b-dc1b9eddcffb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:28.983185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f858bf-bc1f-4972-b29b-dc1b9eddcffb admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:28.984289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f858bf-bc1f-4972-b29b-dc1b9eddcffb admin admin] PATH_INFO: `/v3/users/798c86ea7ce847e6b2dda58ba4af88e2` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:29.005184 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f858bf-bc1f-4972-b29b-dc1b9eddcffb admin admin] RBAC: Authorizing `identity:delete_user(user_id=798c86ea7ce847e6b2dda58ba4af88e2)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:29.013737 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f858bf-bc1f-4972-b29b-dc1b9eddcffb admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:29.168537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-05f858bf-bc1f-4972-b29b-dc1b9eddcffb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '798c86ea7ce847e6b2dda58ba4af88e2'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:29.172528 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-05f858bf-bc1f-4972-b29b-dc1b9eddcffb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '798c86ea7ce847e6b2dda58ba4af88e2'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:29.182027 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-05f858bf-bc1f-4972-b29b-dc1b9eddcffb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '798c86ea7ce847e6b2dda58ba4af88e2'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:29.204291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-05f858bf-bc1f-4972-b29b-dc1b9eddcffb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '798c86ea7ce847e6b2dda58ba4af88e2'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:29.242511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-05f858bf-bc1f-4972-b29b-dc1b9eddcffb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '798c86ea7ce847e6b2dda58ba4af88e2'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:29.281140 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1064/2125] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:32:28 2022] DELETE /identity/v3/users/798c86ea7ce847e6b2dda58ba4af88e2 => generated 0 bytes in 521 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:32:30.242735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc697480-5710-4959-b7e9-6d0a3984cb76 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:30.244731 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-dc697480-5710-4959-b7e9-6d0a3984cb76 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:30.293023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc697480-5710-4959-b7e9-6d0a3984cb76 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:30.356854 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed49191f-7339-4f88-b7df-7fa0f95b24cb None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:30.357377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed49191f-7339-4f88-b7df-7fa0f95b24cb None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:30.357468 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed49191f-7339-4f88-b7df-7fa0f95b24cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:30.390707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-ed49191f-7339-4f88-b7df-7fa0f95b24cb None None] MFA Rules not processed for user `2b4c980bac3e444faa0e6f1a1d270cfa`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:30.392559 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ed49191f-7339-4f88-b7df-7fa0f95b24cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:30.502022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4998ce4c-3158-44cc-91a4-61ec0d4eabb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:30.502703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998ce4c-3158-44cc-91a4-61ec0d4eabb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:30.502980 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998ce4c-3158-44cc-91a4-61ec0d4eabb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:30.503243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998ce4c-3158-44cc-91a4-61ec0d4eabb8 admin admin] PATH_INFO: `/v3/projects/929475f7f7984fda808ef3aac39bbf3f` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:30.504586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4998ce4c-3158-44cc-91a4-61ec0d4eabb8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=929475f7f7984fda808ef3aac39bbf3f)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:30.507669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4998ce4c-3158-44cc-91a4-61ec0d4eabb8 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:30.553304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1065/2126] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:32:30 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:32:30.557147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:30.705394 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-4998ce4c-3158-44cc-91a4-61ec0d4eabb8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '929475f7f7984fda808ef3aac39bbf3f', 'request_id': 'req-4998ce4c-3158-44cc-91a4-61ec0d4eabb8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:30.729319 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-4998ce4c-3158-44cc-91a4-61ec0d4eabb8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '929475f7f7984fda808ef3aac39bbf3f', 'request_id': 'req-4998ce4c-3158-44cc-91a4-61ec0d4eabb8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:30.768665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-4998ce4c-3158-44cc-91a4-61ec0d4eabb8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '929475f7f7984fda808ef3aac39bbf3f', 'request_id': 'req-4998ce4c-3158-44cc-91a4-61ec0d4eabb8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:30.827888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-4998ce4c-3158-44cc-91a4-61ec0d4eabb8 admin admin] The token cache is being invalidate because project 929475f7f7984fda808ef3aac39bbf3f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:32:30.828010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-4998ce4c-3158-44cc-91a4-61ec0d4eabb8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:30.829281 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-4998ce4c-3158-44cc-91a4-61ec0d4eabb8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:30.839783 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1062/2127] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:32:30 2022] DELETE /identity/v3/projects/929475f7f7984fda808ef3aac39bbf3f => generated 0 bytes in 592 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:32:30.840824 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6a2b85-ff51-452a-aef0-91228821d761 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:30.841437 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6a2b85-ff51-452a-aef0-91228821d761 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:30.841934 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6a2b85-ff51-452a-aef0-91228821d761 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:30.917746 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-aa6a2b85-ff51-452a-aef0-91228821d761 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:30.919269 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-aa6a2b85-ff51-452a-aef0-91228821d761 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:31.046526 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:31.121614 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1066/2128] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:32:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:32:31.142748 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9994cfc-1917-4513-971a-6d481136141b None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:31.143329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9994cfc-1917-4513-971a-6d481136141b None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:31.143782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9994cfc-1917-4513-971a-6d481136141b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:31.219431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-a9994cfc-1917-4513-971a-6d481136141b None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:31.222303 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a9994cfc-1917-4513-971a-6d481136141b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:31.253144 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1063/2129] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:32:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:32:31.256751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6a2b85-ff51-452a-aef0-91228821d761 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:31.278928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6a2b85-ff51-452a-aef0-91228821d761 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:31.294026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf4d0dd-fd19-4d66-adc9-1b866446cca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:31.294827 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf4d0dd-fd19-4d66-adc9-1b866446cca2 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:31.295110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf4d0dd-fd19-4d66-adc9-1b866446cca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:31.295384 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf4d0dd-fd19-4d66-adc9-1b866446cca2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:31.296060 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf4d0dd-fd19-4d66-adc9-1b866446cca2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:32:31.300345 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf4d0dd-fd19-4d66-adc9-1b866446cca2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:31.301933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf4d0dd-fd19-4d66-adc9-1b866446cca2 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:31.332942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1067/2130] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:32:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:32:31.335352 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9994cfc-1917-4513-971a-6d481136141b None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:31.350147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9994cfc-1917-4513-971a-6d481136141b None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:31.375061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c49b6f-60b8-4fa5-af39-65f76656d100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:31.376174 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c49b6f-60b8-4fa5-af39-65f76656d100 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:31.376927 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c49b6f-60b8-4fa5-af39-65f76656d100 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:31.377433 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c49b6f-60b8-4fa5-af39-65f76656d100 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:31.378390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c49b6f-60b8-4fa5-af39-65f76656d100 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:31.380854 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c49b6f-60b8-4fa5-af39-65f76656d100 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:31.538976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1064/2131] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:32:31 2022] POST /identity/v3/projects => generated 391 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:32:31.579969 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf4d0dd-fd19-4d66-adc9-1b866446cca2 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:31.602701 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf4d0dd-fd19-4d66-adc9-1b866446cca2 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:31.790233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e20a5bf-949f-48b6-8bce-993c10ad56dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:31.790785 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e20a5bf-949f-48b6-8bce-993c10ad56dd admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:31.791063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e20a5bf-949f-48b6-8bce-993c10ad56dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:31.791333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e20a5bf-949f-48b6-8bce-993c10ad56dd admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:31.791980 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e20a5bf-949f-48b6-8bce-993c10ad56dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:31.794009 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e20a5bf-949f-48b6-8bce-993c10ad56dd admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:31.837899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1068/2132] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:32:31 2022] POST /identity/v3/users => generated 296 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:32:31.842011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c49b6f-60b8-4fa5-af39-65f76656d100 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:31.872135 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c49b6f-60b8-4fa5-af39-65f76656d100 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:31.904361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb901e9b-6c22-4b10-84a1-3475c1b8575b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:31.905129 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb901e9b-6c22-4b10-84a1-3475c1b8575b admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:31.905383 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb901e9b-6c22-4b10-84a1-3475c1b8575b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:31.905691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb901e9b-6c22-4b10-84a1-3475c1b8575b admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:31.906403 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb901e9b-6c22-4b10-84a1-3475c1b8575b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:31.909928 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb901e9b-6c22-4b10-84a1-3475c1b8575b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:31.930432 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1065/2133] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:32:31 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:32:31.934547 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e20a5bf-949f-48b6-8bce-993c10ad56dd admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:31.963671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e20a5bf-949f-48b6-8bce-993c10ad56dd admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:31.971746 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9814cf-ce7e-4529-a65c-3e6968fe1564 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:31.972408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9814cf-ce7e-4529-a65c-3e6968fe1564 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:31.972697 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9814cf-ce7e-4529-a65c-3e6968fe1564 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:31.972976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9814cf-ce7e-4529-a65c-3e6968fe1564 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:31.973641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9814cf-ce7e-4529-a65c-3e6968fe1564 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:31.975269 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9814cf-ce7e-4529-a65c-3e6968fe1564 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:32.031477 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb901e9b-6c22-4b10-84a1-3475c1b8575b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:32.042148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1069/2134] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:32:31 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:32:32.049068 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb901e9b-6c22-4b10-84a1-3475c1b8575b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:32.065182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ddf6dad-a527-4668-a82b-aad45735dd11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:32.065884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ddf6dad-a527-4668-a82b-aad45735dd11 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:32.066169 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ddf6dad-a527-4668-a82b-aad45735dd11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:32.066439 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ddf6dad-a527-4668-a82b-aad45735dd11 admin admin] PATH_INFO: `/v3/projects/75521a0d89554160a2fccc7b789ac83e/users/18b1494b78f644bcb30b6c5f8dc2a422/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:32.119538 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ddf6dad-a527-4668-a82b-aad45735dd11 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75521a0d89554160a2fccc7b789ac83e, user_id=18b1494b78f644bcb30b6c5f8dc2a422, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:32.122561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ddf6dad-a527-4668-a82b-aad45735dd11 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:32.165831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13af6ea9-cda4-4ae3-a0b7-668a4b4a9d11 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:32.166225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13af6ea9-cda4-4ae3-a0b7-668a4b4a9d11 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:32.166410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13af6ea9-cda4-4ae3-a0b7-668a4b4a9d11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:32.169254 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1066/2135] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:32:32 2022] PUT /identity/v3/projects/75521a0d89554160a2fccc7b789ac83e/users/18b1494b78f644bcb30b6c5f8dc2a422/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:32:32.240130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-13af6ea9-cda4-4ae3-a0b7-668a4b4a9d11 None None] MFA Rules not processed for user `18b1494b78f644bcb30b6c5f8dc2a422`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:32.243718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-13af6ea9-cda4-4ae3-a0b7-668a4b4a9d11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:32.565864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1070/2136] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:32:32 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 401 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:32:32.570344 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ddf6dad-a527-4668-a82b-aad45735dd11 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:32.705644 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ddf6dad-a527-4668-a82b-aad45735dd11 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:32.713994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509c0bfd-59c0-47ad-a167-272666b79916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:32.714640 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509c0bfd-59c0-47ad-a167-272666b79916 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:32.715318 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509c0bfd-59c0-47ad-a167-272666b79916 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:32.715600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509c0bfd-59c0-47ad-a167-272666b79916 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:32.716449 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509c0bfd-59c0-47ad-a167-272666b79916 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:32.718332 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509c0bfd-59c0-47ad-a167-272666b79916 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:32.782734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1067/2137] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:32:32 2022] POST /identity/v3/projects => generated 393 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:32:32.786462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13af6ea9-cda4-4ae3-a0b7-668a4b4a9d11 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:32.807748 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13af6ea9-cda4-4ae3-a0b7-668a4b4a9d11 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:32.962065 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:33.066080 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df912491-f269-4db4-bf53-4bc5cfabdc6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:33.067649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df912491-f269-4db4-bf53-4bc5cfabdc6f admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:33.068188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df912491-f269-4db4-bf53-4bc5cfabdc6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:33.068531 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df912491-f269-4db4-bf53-4bc5cfabdc6f admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:33.069948 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df912491-f269-4db4-bf53-4bc5cfabdc6f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:33.072396 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df912491-f269-4db4-bf53-4bc5cfabdc6f admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:33.109748 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1071/2138] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:32:32 2022] POST /identity/v3/users => generated 303 bytes in 324 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:32:33.114681 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509c0bfd-59c0-47ad-a167-272666b79916 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:33.137055 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509c0bfd-59c0-47ad-a167-272666b79916 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:33.147794 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5734b093-6b94-4060-8ebb-6624c6d4adfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:33.147794 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5734b093-6b94-4060-8ebb-6624c6d4adfa admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:33.149004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5734b093-6b94-4060-8ebb-6624c6d4adfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:33.149295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5734b093-6b94-4060-8ebb-6624c6d4adfa admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:33.150033 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5734b093-6b94-4060-8ebb-6624c6d4adfa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:33.151745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5734b093-6b94-4060-8ebb-6624c6d4adfa admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:33.173549 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1068/2139] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:32:33 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:32:33.177701 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df912491-f269-4db4-bf53-4bc5cfabdc6f admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:33.195421 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df912491-f269-4db4-bf53-4bc5cfabdc6f admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:33.204991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c453433-5a4d-44b8-ad74-6db31fa1dd79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:33.207268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c453433-5a4d-44b8-ad74-6db31fa1dd79 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:33.207576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c453433-5a4d-44b8-ad74-6db31fa1dd79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:33.207843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c453433-5a4d-44b8-ad74-6db31fa1dd79 admin admin] PATH_INFO: `/v3/projects/27baf095429145cab4744d3cb19f4a09/users/40a3baf2d6794ea99727c9a389e04686/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:33.229164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c453433-5a4d-44b8-ad74-6db31fa1dd79 admin admin] RBAC: Authorizing `identity:create_grant(project_id=27baf095429145cab4744d3cb19f4a09, user_id=40a3baf2d6794ea99727c9a389e04686, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:33.232023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c453433-5a4d-44b8-ad74-6db31fa1dd79 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:33.266768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1072/2140] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:32:33 2022] PUT /identity/v3/projects/27baf095429145cab4744d3cb19f4a09/users/40a3baf2d6794ea99727c9a389e04686/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:32:33.270630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5734b093-6b94-4060-8ebb-6624c6d4adfa admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:33.286646 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5734b093-6b94-4060-8ebb-6624c6d4adfa admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:33.304042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:33.478018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1663efb-487b-421c-8bcd-09a40279af01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:33.478659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1663efb-487b-421c-8bcd-09a40279af01 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:33.478976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1663efb-487b-421c-8bcd-09a40279af01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:33.479221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1663efb-487b-421c-8bcd-09a40279af01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:33.480248 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1663efb-487b-421c-8bcd-09a40279af01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:33.495369 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1663efb-487b-421c-8bcd-09a40279af01 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:33.525017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1069/2141] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:32:33 2022] GET /identity/v3/roles => generated 1420 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:32:33.527716 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c453433-5a4d-44b8-ad74-6db31fa1dd79 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:33.546872 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c453433-5a4d-44b8-ad74-6db31fa1dd79 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:33.553094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b7b37e-7687-4c9f-a7c2-48cd3470f9c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:33.553877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b7b37e-7687-4c9f-a7c2-48cd3470f9c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:33.554164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b7b37e-7687-4c9f-a7c2-48cd3470f9c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:33.554446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b7b37e-7687-4c9f-a7c2-48cd3470f9c1 admin admin] PATH_INFO: `/v3/projects/27baf095429145cab4744d3cb19f4a09/users/40a3baf2d6794ea99727c9a389e04686/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:33.556786 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b7b37e-7687-4c9f-a7c2-48cd3470f9c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=27baf095429145cab4744d3cb19f4a09, user_id=40a3baf2d6794ea99727c9a389e04686, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:33.559771 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b7b37e-7687-4c9f-a7c2-48cd3470f9c1 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:33.591309 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1073/2142] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:32:33 2022] PUT /identity/v3/projects/27baf095429145cab4744d3cb19f4a09/users/40a3baf2d6794ea99727c9a389e04686/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:32:33.604423 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc7a16b-5dbc-4218-9d39-ee8323d6f25c None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:33.604975 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc7a16b-5dbc-4218-9d39-ee8323d6f25c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:33.604975 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc7a16b-5dbc-4218-9d39-ee8323d6f25c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:33.690776 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-4cc7a16b-5dbc-4218-9d39-ee8323d6f25c None None] MFA Rules not processed for user `40a3baf2d6794ea99727c9a389e04686`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:33.693977 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-4cc7a16b-5dbc-4218-9d39-ee8323d6f25c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:34.018637 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1070/2143] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:32:33 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 412 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:32:34.029481 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b7b37e-7687-4c9f-a7c2-48cd3470f9c1 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:34.031578 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-30b7b37e-7687-4c9f-a7c2-48cd3470f9c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:34.396403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b7b37e-7687-4c9f-a7c2-48cd3470f9c1 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:34.396403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd0b8da-1be4-446b-a528-20ab9fd17e8a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:34.396403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd0b8da-1be4-446b-a528-20ab9fd17e8a service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:34.396403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd0b8da-1be4-446b-a528-20ab9fd17e8a service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:34.397011 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd0b8da-1be4-446b-a528-20ab9fd17e8a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:34.418533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd0b8da-1be4-446b-a528-20ab9fd17e8a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:34.420487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd0b8da-1be4-446b-a528-20ab9fd17e8a service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:34.436215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1074/2144] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:32:34 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:32:34.555094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc7a16b-5dbc-4218-9d39-ee8323d6f25c None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:34.556723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-4cc7a16b-5dbc-4218-9d39-ee8323d6f25c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:34.594388 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc7a16b-5dbc-4218-9d39-ee8323d6f25c None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:34.615383 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bbeb5de-dea9-4004-9ee4-a5b7e6167dda service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:34.616130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbeb5de-dea9-4004-9ee4-a5b7e6167dda service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:34.616250 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbeb5de-dea9-4004-9ee4-a5b7e6167dda service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:34.616586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbeb5de-dea9-4004-9ee4-a5b7e6167dda service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:34.796583 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bbeb5de-dea9-4004-9ee4-a5b7e6167dda service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:34.800451 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bbeb5de-dea9-4004-9ee4-a5b7e6167dda service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:34.847689 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1071/2145] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:32:34 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:32:39.786016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd0b8da-1be4-446b-a528-20ab9fd17e8a service cinder] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:39.978530 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd0b8da-1be4-446b-a528-20ab9fd17e8a service cinder] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:39.984847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212f91b8-6c2f-4dbb-8481-b123ecf57abe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:39.986165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212f91b8-6c2f-4dbb-8481-b123ecf57abe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:39.986520 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212f91b8-6c2f-4dbb-8481-b123ecf57abe admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:39.986896 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212f91b8-6c2f-4dbb-8481-b123ecf57abe admin admin] PATH_INFO: `/v3/users/18b1494b78f644bcb30b6c5f8dc2a422` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:39.988149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212f91b8-6c2f-4dbb-8481-b123ecf57abe admin admin] RBAC: Authorizing `identity:delete_user(user_id=18b1494b78f644bcb30b6c5f8dc2a422)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:39.994979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212f91b8-6c2f-4dbb-8481-b123ecf57abe admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:40.093549 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:40.138914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-212f91b8-6c2f-4dbb-8481-b123ecf57abe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '18b1494b78f644bcb30b6c5f8dc2a422'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:40.140147 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-212f91b8-6c2f-4dbb-8481-b123ecf57abe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '18b1494b78f644bcb30b6c5f8dc2a422'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:40.140815 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-212f91b8-6c2f-4dbb-8481-b123ecf57abe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '18b1494b78f644bcb30b6c5f8dc2a422'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:40.154413 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-212f91b8-6c2f-4dbb-8481-b123ecf57abe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '18b1494b78f644bcb30b6c5f8dc2a422'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:40.178638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-212f91b8-6c2f-4dbb-8481-b123ecf57abe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '18b1494b78f644bcb30b6c5f8dc2a422'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:40.197703 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1075/2146] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:32:39 2022] DELETE /identity/v3/users/18b1494b78f644bcb30b6c5f8dc2a422 => generated 0 bytes in 418 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:32:40.208766 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ed0384-1409-45b5-a5cd-243c5c3c8d0c None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:40.209935 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ed0384-1409-45b5-a5cd-243c5c3c8d0c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:40.210373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ed0384-1409-45b5-a5cd-243c5c3c8d0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:40.267035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-45ed0384-1409-45b5-a5cd-243c5c3c8d0c None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:40.270548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-45ed0384-1409-45b5-a5cd-243c5c3c8d0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:40.353692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:40.464379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1072/2147] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:32:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 258 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:32:40.476799 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212f91b8-6c2f-4dbb-8481-b123ecf57abe admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:40.478520 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-212f91b8-6c2f-4dbb-8481-b123ecf57abe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:40.665735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212f91b8-6c2f-4dbb-8481-b123ecf57abe admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:40.670279 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd24519-5237-43ee-900f-2fc1b935e7f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:40.670944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd24519-5237-43ee-900f-2fc1b935e7f3 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:40.671284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd24519-5237-43ee-900f-2fc1b935e7f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:40.671846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd24519-5237-43ee-900f-2fc1b935e7f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:40.687235 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd24519-5237-43ee-900f-2fc1b935e7f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:40.694902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd24519-5237-43ee-900f-2fc1b935e7f3 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:40.720847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1076/2148] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:32:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:32:41.517187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ed0384-1409-45b5-a5cd-243c5c3c8d0c None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:41.518950 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-45ed0384-1409-45b5-a5cd-243c5c3c8d0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:41.538905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ed0384-1409-45b5-a5cd-243c5c3c8d0c None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:41.555342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830625ff-7b13-42a3-801a-4a4517678361 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:41.556362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830625ff-7b13-42a3-801a-4a4517678361 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:41.556913 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830625ff-7b13-42a3-801a-4a4517678361 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:41.557295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830625ff-7b13-42a3-801a-4a4517678361 admin admin] PATH_INFO: `/v3/projects/75521a0d89554160a2fccc7b789ac83e` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:41.559416 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830625ff-7b13-42a3-801a-4a4517678361 admin admin] RBAC: Authorizing `identity:delete_project(project_id=75521a0d89554160a2fccc7b789ac83e)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:41.566288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830625ff-7b13-42a3-801a-4a4517678361 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:41.699887 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-830625ff-7b13-42a3-801a-4a4517678361 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75521a0d89554160a2fccc7b789ac83e', 'request_id': 'req-830625ff-7b13-42a3-801a-4a4517678361'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:41.714026 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-830625ff-7b13-42a3-801a-4a4517678361 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75521a0d89554160a2fccc7b789ac83e', 'request_id': 'req-830625ff-7b13-42a3-801a-4a4517678361'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:41.732492 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-830625ff-7b13-42a3-801a-4a4517678361 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75521a0d89554160a2fccc7b789ac83e', 'request_id': 'req-830625ff-7b13-42a3-801a-4a4517678361'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:41.765002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-830625ff-7b13-42a3-801a-4a4517678361 admin admin] The token cache is being invalidate because project 75521a0d89554160a2fccc7b789ac83e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:32:41.765302 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-830625ff-7b13-42a3-801a-4a4517678361 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:41.767826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-830625ff-7b13-42a3-801a-4a4517678361 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:41.790516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1073/2149] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:32:41 2022] DELETE /identity/v3/projects/75521a0d89554160a2fccc7b789ac83e => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:32:41.791835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd24519-5237-43ee-900f-2fc1b935e7f3 service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:41.803649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd24519-5237-43ee-900f-2fc1b935e7f3 service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:42.016530 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f658fa96-da39-4492-be63-96476a854ad2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:42.017762 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f658fa96-da39-4492-be63-96476a854ad2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:42.018270 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f658fa96-da39-4492-be63-96476a854ad2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:42.018811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f658fa96-da39-4492-be63-96476a854ad2 admin admin] PATH_INFO: `/v3/users/40a3baf2d6794ea99727c9a389e04686` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:42.020568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f658fa96-da39-4492-be63-96476a854ad2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=40a3baf2d6794ea99727c9a389e04686)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:42.023270 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f658fa96-da39-4492-be63-96476a854ad2 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:42.174656 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-f658fa96-da39-4492-be63-96476a854ad2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '40a3baf2d6794ea99727c9a389e04686'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:42.176030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-f658fa96-da39-4492-be63-96476a854ad2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '40a3baf2d6794ea99727c9a389e04686'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:42.177297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-f658fa96-da39-4492-be63-96476a854ad2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '40a3baf2d6794ea99727c9a389e04686'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:42.198803 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-f658fa96-da39-4492-be63-96476a854ad2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '40a3baf2d6794ea99727c9a389e04686'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:42.202079 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:42.231482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d36edb-d7c6-4cff-8bf4-65c4416448ef None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:42.231762 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d36edb-d7c6-4cff-8bf4-65c4416448ef None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:42.232038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d36edb-d7c6-4cff-8bf4-65c4416448ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:42.234185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-f658fa96-da39-4492-be63-96476a854ad2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '40a3baf2d6794ea99727c9a389e04686'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:42.258701 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1077/2150] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:32:41 2022] DELETE /identity/v3/users/40a3baf2d6794ea99727c9a389e04686 => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:32:42.274109 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-50d36edb-d7c6-4cff-8bf4-65c4416448ef None None] MFA Rules not processed for user `34afa4aac10c4a018ca05a0ba34c940a`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:42.276144 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-50d36edb-d7c6-4cff-8bf4-65c4416448ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:42.394846 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1074/2151] 10.222.0.64 () {58 vars in 967 bytes} [Tue Oct 4 14:32:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Oct 04 14:32:42.425921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f658fa96-da39-4492-be63-96476a854ad2 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:42.427775 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-f658fa96-da39-4492-be63-96476a854ad2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:42.650928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f658fa96-da39-4492-be63-96476a854ad2 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:42.661839 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb3db03-69bd-4966-93fc-1e7701168a83 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5980e353164c4873bcdd2af847e0a1f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:42.663240 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb3db03-69bd-4966-93fc-1e7701168a83 service swift] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:42.663704 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb3db03-69bd-4966-93fc-1e7701168a83 service swift] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:42.664158 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb3db03-69bd-4966-93fc-1e7701168a83 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:42.678786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d36edb-d7c6-4cff-8bf4-65c4416448ef None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:42.680990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-50d36edb-d7c6-4cff-8bf4-65c4416448ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:42.697574 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb3db03-69bd-4966-93fc-1e7701168a83 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:42.699886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb3db03-69bd-4966-93fc-1e7701168a83 service swift] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:42.733343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1078/2152] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Oct 4 14:32:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 14:32:42.827569 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:42.840265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d36edb-d7c6-4cff-8bf4-65c4416448ef None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:42.859506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749e496f-885b-4eeb-9c5b-07721a0bb16c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '436deb3d60c54a799700986519b2ef68', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:42.860238 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749e496f-885b-4eeb-9c5b-07721a0bb16c service placement] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:42.860568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749e496f-885b-4eeb-9c5b-07721a0bb16c service placement] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:42.860842 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749e496f-885b-4eeb-9c5b-07721a0bb16c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:42.866956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-749e496f-885b-4eeb-9c5b-07721a0bb16c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:42.898895 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749e496f-885b-4eeb-9c5b-07721a0bb16c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:42.903991 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749e496f-885b-4eeb-9c5b-07721a0bb16c service placement] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:42.941963 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1075/2153] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:32:42 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 263 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:32:43.338345 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb3db03-69bd-4966-93fc-1e7701168a83 service swift] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:43.340748 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ffb3db03-69bd-4966-93fc-1e7701168a83 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:43.444894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749e496f-885b-4eeb-9c5b-07721a0bb16c service placement] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:43.446431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-749e496f-885b-4eeb-9c5b-07721a0bb16c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:43.689775 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb3db03-69bd-4966-93fc-1e7701168a83 service swift] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:43.702127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811f45ce-6685-4a94-8f38-a502dc9d8a31 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:43.702791 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811f45ce-6685-4a94-8f38-a502dc9d8a31 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:43.703400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811f45ce-6685-4a94-8f38-a502dc9d8a31 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:43.705447 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811f45ce-6685-4a94-8f38-a502dc9d8a31 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:43.709428 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-811f45ce-6685-4a94-8f38-a502dc9d8a31 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:43.711027 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749e496f-885b-4eeb-9c5b-07721a0bb16c service placement] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:43.720429 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6047adfe-cbb7-4bc6-a08f-a4de628b4756 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:43.722111 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6047adfe-cbb7-4bc6-a08f-a4de628b4756 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:43.722426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6047adfe-cbb7-4bc6-a08f-a4de628b4756 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:43.722771 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6047adfe-cbb7-4bc6-a08f-a4de628b4756 admin admin] PATH_INFO: `/v3/projects/27baf095429145cab4744d3cb19f4a09` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:43.724220 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6047adfe-cbb7-4bc6-a08f-a4de628b4756 admin admin] RBAC: Authorizing `identity:delete_project(project_id=27baf095429145cab4744d3cb19f4a09)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:43.728019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6047adfe-cbb7-4bc6-a08f-a4de628b4756 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:43.889635 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811f45ce-6685-4a94-8f38-a502dc9d8a31 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:43.891843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811f45ce-6685-4a94-8f38-a502dc9d8a31 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:43.925299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1079/2154] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:32:43 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 587 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:32:43.940979 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6047adfe-cbb7-4bc6-a08f-a4de628b4756 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '27baf095429145cab4744d3cb19f4a09', 'request_id': 'req-6047adfe-cbb7-4bc6-a08f-a4de628b4756'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:43.959175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6047adfe-cbb7-4bc6-a08f-a4de628b4756 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '27baf095429145cab4744d3cb19f4a09', 'request_id': 'req-6047adfe-cbb7-4bc6-a08f-a4de628b4756'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:44.010507 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6047adfe-cbb7-4bc6-a08f-a4de628b4756 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '27baf095429145cab4744d3cb19f4a09', 'request_id': 'req-6047adfe-cbb7-4bc6-a08f-a4de628b4756'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:44.036575 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6047adfe-cbb7-4bc6-a08f-a4de628b4756 admin admin] The token cache is being invalidate because project 27baf095429145cab4744d3cb19f4a09 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:32:44.036692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6047adfe-cbb7-4bc6-a08f-a4de628b4756 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:44.040453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6047adfe-cbb7-4bc6-a08f-a4de628b4756 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:44.043732 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1076/2155] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:32:43 2022] DELETE /identity/v3/projects/27baf095429145cab4744d3cb19f4a09 => generated 0 bytes in 599 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:32:44.049706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4098e309-a94c-4548-bbcc-e4b37f07a982 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:44.050330 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4098e309-a94c-4548-bbcc-e4b37f07a982 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:44.050956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4098e309-a94c-4548-bbcc-e4b37f07a982 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:44.080174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-4098e309-a94c-4548-bbcc-e4b37f07a982 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:44.081643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-4098e309-a94c-4548-bbcc-e4b37f07a982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:44.088978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:44.347398 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1080/2156] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:32:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:32:44.356141 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d6ea29-ad92-43c1-9a45-75ea5e2c5a38 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:44.356456 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d6ea29-ad92-43c1-9a45-75ea5e2c5a38 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:44.356916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d6ea29-ad92-43c1-9a45-75ea5e2c5a38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:44.413086 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-77d6ea29-ad92-43c1-9a45-75ea5e2c5a38 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:44.416392 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-77d6ea29-ad92-43c1-9a45-75ea5e2c5a38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:44.432542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1077/2157] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:32:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:32:44.436682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4098e309-a94c-4548-bbcc-e4b37f07a982 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:44.452157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4098e309-a94c-4548-bbcc-e4b37f07a982 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:44.483000 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300d24b0-5307-4ad4-87c4-411ade33096a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:44.483632 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300d24b0-5307-4ad4-87c4-411ade33096a admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:44.484106 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300d24b0-5307-4ad4-87c4-411ade33096a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:44.484577 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300d24b0-5307-4ad4-87c4-411ade33096a admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:44.485488 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300d24b0-5307-4ad4-87c4-411ade33096a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:32:44.486039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300d24b0-5307-4ad4-87c4-411ade33096a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:44.487928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300d24b0-5307-4ad4-87c4-411ade33096a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:44.519682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1081/2158] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:32:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:32:44.524234 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d6ea29-ad92-43c1-9a45-75ea5e2c5a38 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:44.555000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d6ea29-ad92-43c1-9a45-75ea5e2c5a38 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:44.650945 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d15b94-f0c7-4ef6-889c-f6fb5dc2691b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:44.651553 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d15b94-f0c7-4ef6-889c-f6fb5dc2691b admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:44.654707 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d15b94-f0c7-4ef6-889c-f6fb5dc2691b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:44.655091 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d15b94-f0c7-4ef6-889c-f6fb5dc2691b admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:44.655937 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d15b94-f0c7-4ef6-889c-f6fb5dc2691b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:44.669006 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d15b94-f0c7-4ef6-889c-f6fb5dc2691b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:44.747525 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1078/2159] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:32:44 2022] POST /identity/v3/projects => generated 377 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:32:44.758503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300d24b0-5307-4ad4-87c4-411ade33096a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:44.788146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300d24b0-5307-4ad4-87c4-411ade33096a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:44.933588 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe52b77-e5ba-4732-a5b8-a363b310a7b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:44.934629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe52b77-e5ba-4732-a5b8-a363b310a7b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:44.935128 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe52b77-e5ba-4732-a5b8-a363b310a7b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:44.935713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe52b77-e5ba-4732-a5b8-a363b310a7b5 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:44.936613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe52b77-e5ba-4732-a5b8-a363b310a7b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:44.939712 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe52b77-e5ba-4732-a5b8-a363b310a7b5 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:44.973706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1082/2160] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:32:44 2022] POST /identity/v3/users => generated 289 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:32:44.974951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d15b94-f0c7-4ef6-889c-f6fb5dc2691b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:44.988501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d15b94-f0c7-4ef6-889c-f6fb5dc2691b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:45.006460 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6332b037-9fd8-4ed9-932a-7513eea39715 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:45.007271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6332b037-9fd8-4ed9-932a-7513eea39715 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:45.007550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6332b037-9fd8-4ed9-932a-7513eea39715 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:45.007858 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6332b037-9fd8-4ed9-932a-7513eea39715 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:45.008739 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6332b037-9fd8-4ed9-932a-7513eea39715 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:45.011231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6332b037-9fd8-4ed9-932a-7513eea39715 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:45.042958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1079/2161] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:32:44 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:32:45.044661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe52b77-e5ba-4732-a5b8-a363b310a7b5 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:45.065161 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe52b77-e5ba-4732-a5b8-a363b310a7b5 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:45.090548 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac5873e-c6fb-4112-993e-9b7cdf3a7254 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:45.092510 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac5873e-c6fb-4112-993e-9b7cdf3a7254 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:45.096122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac5873e-c6fb-4112-993e-9b7cdf3a7254 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:45.096593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac5873e-c6fb-4112-993e-9b7cdf3a7254 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:45.099557 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac5873e-c6fb-4112-993e-9b7cdf3a7254 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:45.107592 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac5873e-c6fb-4112-993e-9b7cdf3a7254 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:45.135707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1083/2162] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:32:45 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:32:45.137072 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6332b037-9fd8-4ed9-932a-7513eea39715 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:45.176371 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6332b037-9fd8-4ed9-932a-7513eea39715 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:45.195856 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42d6083-dcd5-45d6-a096-606882c8b956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:45.196565 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42d6083-dcd5-45d6-a096-606882c8b956 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:45.196849 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42d6083-dcd5-45d6-a096-606882c8b956 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:45.197120 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42d6083-dcd5-45d6-a096-606882c8b956 admin admin] PATH_INFO: `/v3/projects/c9fc3a9be06c446db32b810b6a614dd5/users/11261fd586e440749ba943fc2255c2a4/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:45.240834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42d6083-dcd5-45d6-a096-606882c8b956 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c9fc3a9be06c446db32b810b6a614dd5, user_id=11261fd586e440749ba943fc2255c2a4, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:45.244157 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42d6083-dcd5-45d6-a096-606882c8b956 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:45.275653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1080/2163] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:32:45 2022] PUT /identity/v3/projects/c9fc3a9be06c446db32b810b6a614dd5/users/11261fd586e440749ba943fc2255c2a4/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:32:45.288100 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d568d1-8e69-4e3b-b00f-05a2263ad18f None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:45.288595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d568d1-8e69-4e3b-b00f-05a2263ad18f None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:45.288951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d568d1-8e69-4e3b-b00f-05a2263ad18f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:45.390376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-27d568d1-8e69-4e3b-b00f-05a2263ad18f None None] MFA Rules not processed for user `11261fd586e440749ba943fc2255c2a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:45.392056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-27d568d1-8e69-4e3b-b00f-05a2263ad18f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:45.614612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1084/2164] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:32:45 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 328 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:32:46.114436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42d6083-dcd5-45d6-a096-606882c8b956 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:46.118182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f42d6083-dcd5-45d6-a096-606882c8b956 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:46.145426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:46.221059 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42d6083-dcd5-45d6-a096-606882c8b956 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:46.226170 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4524426e-706c-4efa-ba13-f4acfc9b03bc service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35a5a4ef40204e158649ca25583bbde7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:46.226912 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4524426e-706c-4efa-ba13-f4acfc9b03bc service glance] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:46.227474 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4524426e-706c-4efa-ba13-f4acfc9b03bc service glance] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:46.228068 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4524426e-706c-4efa-ba13-f4acfc9b03bc service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:46.251639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4524426e-706c-4efa-ba13-f4acfc9b03bc service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:46.253743 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4524426e-706c-4efa-ba13-f4acfc9b03bc service glance] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:46.266873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1081/2165] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:32:46 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:32:46.634952 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d568d1-8e69-4e3b-b00f-05a2263ad18f None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:46.637503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-27d568d1-8e69-4e3b-b00f-05a2263ad18f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:46.667719 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:46.770663 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d568d1-8e69-4e3b-b00f-05a2263ad18f None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:46.777627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3e5ea5-07a8-471e-9172-3d0d803faa99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:46.778280 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3e5ea5-07a8-471e-9172-3d0d803faa99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:46.778560 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3e5ea5-07a8-471e-9172-3d0d803faa99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:46.778834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3e5ea5-07a8-471e-9172-3d0d803faa99 admin admin] PATH_INFO: `/v3/users/b3bd51e1acb548ea957ffb57cb09f3f4` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:46.781502 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3e5ea5-07a8-471e-9172-3d0d803faa99 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b3bd51e1acb548ea957ffb57cb09f3f4)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:46.784127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3e5ea5-07a8-471e-9172-3d0d803faa99 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:46.857794 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-bc3e5ea5-07a8-471e-9172-3d0d803faa99 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b3bd51e1acb548ea957ffb57cb09f3f4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:46.858515 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-bc3e5ea5-07a8-471e-9172-3d0d803faa99 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b3bd51e1acb548ea957ffb57cb09f3f4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:46.859341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-bc3e5ea5-07a8-471e-9172-3d0d803faa99 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b3bd51e1acb548ea957ffb57cb09f3f4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:46.872957 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-bc3e5ea5-07a8-471e-9172-3d0d803faa99 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b3bd51e1acb548ea957ffb57cb09f3f4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:46.894686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-bc3e5ea5-07a8-471e-9172-3d0d803faa99 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b3bd51e1acb548ea957ffb57cb09f3f4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:46.915750 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1085/2166] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:32:46 2022] DELETE /identity/v3/users/b3bd51e1acb548ea957ffb57cb09f3f4 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:32:47.647604 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4524426e-706c-4efa-ba13-f4acfc9b03bc service glance] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:47.671683 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4524426e-706c-4efa-ba13-f4acfc9b03bc service glance] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:47.841263 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8775998-1b3d-49a2-9a45-9c9de8d7b86b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:47.841919 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8775998-1b3d-49a2-9a45-9c9de8d7b86b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:47.842200 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8775998-1b3d-49a2-9a45-9c9de8d7b86b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:47.842475 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8775998-1b3d-49a2-9a45-9c9de8d7b86b admin admin] PATH_INFO: `/v3/projects/ff877a220f9443fb93fde985a653c999` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:47.843710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8775998-1b3d-49a2-9a45-9c9de8d7b86b admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff877a220f9443fb93fde985a653c999)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:47.845926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8775998-1b3d-49a2-9a45-9c9de8d7b86b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:47.912181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b8775998-1b3d-49a2-9a45-9c9de8d7b86b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff877a220f9443fb93fde985a653c999', 'request_id': 'req-b8775998-1b3d-49a2-9a45-9c9de8d7b86b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:47.922724 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b8775998-1b3d-49a2-9a45-9c9de8d7b86b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff877a220f9443fb93fde985a653c999', 'request_id': 'req-b8775998-1b3d-49a2-9a45-9c9de8d7b86b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:47.930705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b8775998-1b3d-49a2-9a45-9c9de8d7b86b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff877a220f9443fb93fde985a653c999', 'request_id': 'req-b8775998-1b3d-49a2-9a45-9c9de8d7b86b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:47.951501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b8775998-1b3d-49a2-9a45-9c9de8d7b86b admin admin] The token cache is being invalidate because project ff877a220f9443fb93fde985a653c999 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:32:47.952807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b8775998-1b3d-49a2-9a45-9c9de8d7b86b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:47.953827 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b8775998-1b3d-49a2-9a45-9c9de8d7b86b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:32:47.957160 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1082/2167] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:32:47 2022] DELETE /identity/v3/projects/ff877a220f9443fb93fde985a653c999 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:32:47.962083 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f38be90-f70d-4ff9-a5a9-e484945beeca None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:47.962635 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f38be90-f70d-4ff9-a5a9-e484945beeca None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:47.962863 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f38be90-f70d-4ff9-a5a9-e484945beeca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:48.004812 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-1f38be90-f70d-4ff9-a5a9-e484945beeca None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:48.006206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-1f38be90-f70d-4ff9-a5a9-e484945beeca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:48.155850 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1086/2168] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:32:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:32:48.169188 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19771dc0-86fb-4b5c-926f-01bd744dfd68 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:48.169895 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19771dc0-86fb-4b5c-926f-01bd744dfd68 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:48.170386 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19771dc0-86fb-4b5c-926f-01bd744dfd68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:48.196981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-19771dc0-86fb-4b5c-926f-01bd744dfd68 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:48.199116 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-19771dc0-86fb-4b5c-926f-01bd744dfd68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:48.216365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1083/2169] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:32:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:32:48.221336 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f38be90-f70d-4ff9-a5a9-e484945beeca None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:48.243976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f38be90-f70d-4ff9-a5a9-e484945beeca None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:48.251073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a076b8-03da-4ebd-8f94-a1adfae414c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:48.251962 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a076b8-03da-4ebd-8f94-a1adfae414c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:48.252782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a076b8-03da-4ebd-8f94-a1adfae414c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:48.253270 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a076b8-03da-4ebd-8f94-a1adfae414c8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:48.254066 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a076b8-03da-4ebd-8f94-a1adfae414c8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:32:48.254418 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a076b8-03da-4ebd-8f94-a1adfae414c8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:48.256770 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a076b8-03da-4ebd-8f94-a1adfae414c8 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:48.278787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1087/2170] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:32:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:32:48.280663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19771dc0-86fb-4b5c-926f-01bd744dfd68 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:48.301607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19771dc0-86fb-4b5c-926f-01bd744dfd68 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:48.322835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edb48a4-d31e-430d-8634-09a031cd7432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:48.323556 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edb48a4-d31e-430d-8634-09a031cd7432 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:48.324058 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edb48a4-d31e-430d-8634-09a031cd7432 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:48.325794 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edb48a4-d31e-430d-8634-09a031cd7432 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:48.326630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edb48a4-d31e-430d-8634-09a031cd7432 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:48.329971 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edb48a4-d31e-430d-8634-09a031cd7432 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:48.393536 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1084/2171] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:32:48 2022] POST /identity/v3/projects => generated 373 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:32:48.401789 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a076b8-03da-4ebd-8f94-a1adfae414c8 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:48.419264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a076b8-03da-4ebd-8f94-a1adfae414c8 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:48.431068 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:48.536586 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f235f480-11dd-4694-b6be-e655d55ab64e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:48.537813 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f235f480-11dd-4694-b6be-e655d55ab64e admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:48.538100 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f235f480-11dd-4694-b6be-e655d55ab64e admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:48.538373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f235f480-11dd-4694-b6be-e655d55ab64e admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:48.539208 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f235f480-11dd-4694-b6be-e655d55ab64e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:48.541350 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f235f480-11dd-4694-b6be-e655d55ab64e admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:48.574159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1088/2172] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:32:48 2022] POST /identity/v3/users => generated 287 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:32:48.579001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edb48a4-d31e-430d-8634-09a031cd7432 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:48.596075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edb48a4-d31e-430d-8634-09a031cd7432 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:48.613274 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b04331-9106-4212-a713-1d887392dfb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:48.614308 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b04331-9106-4212-a713-1d887392dfb7 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:48.614703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b04331-9106-4212-a713-1d887392dfb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:48.615700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b04331-9106-4212-a713-1d887392dfb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:48.618382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b04331-9106-4212-a713-1d887392dfb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:48.621677 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b04331-9106-4212-a713-1d887392dfb7 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:48.635753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1085/2173] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:32:48 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:32:48.639837 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f235f480-11dd-4694-b6be-e655d55ab64e admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:48.664340 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f235f480-11dd-4694-b6be-e655d55ab64e admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:48.682974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502ae546-0945-4629-8717-a746a971d704 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:48.683633 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502ae546-0945-4629-8717-a746a971d704 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:48.683916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502ae546-0945-4629-8717-a746a971d704 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:48.684194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502ae546-0945-4629-8717-a746a971d704 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:48.685070 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502ae546-0945-4629-8717-a746a971d704 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:48.686756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502ae546-0945-4629-8717-a746a971d704 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:48.724170 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b04331-9106-4212-a713-1d887392dfb7 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:48.733767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1089/2174] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:32:48 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:32:48.741526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b04331-9106-4212-a713-1d887392dfb7 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:48.763531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73712c8-22cb-44ad-af4a-51cde1292aea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:48.766002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73712c8-22cb-44ad-af4a-51cde1292aea admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:48.766440 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73712c8-22cb-44ad-af4a-51cde1292aea admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:48.766939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73712c8-22cb-44ad-af4a-51cde1292aea admin admin] PATH_INFO: `/v3/projects/f5980ace2a72461689bfb87d48571e72/users/b03bae15d7bf43bb871c1a2eed6ae09a/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:48.782964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:32:48.807431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73712c8-22cb-44ad-af4a-51cde1292aea admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5980ace2a72461689bfb87d48571e72, user_id=b03bae15d7bf43bb871c1a2eed6ae09a, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:48.816841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73712c8-22cb-44ad-af4a-51cde1292aea admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:48.856078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1086/2175] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:32:48 2022] PUT /identity/v3/projects/f5980ace2a72461689bfb87d48571e72/users/b03bae15d7bf43bb871c1a2eed6ae09a/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:32:48.857639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9000870d-d952-4f2e-a0e6-8716424f4c03 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:48.858035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9000870d-d952-4f2e-a0e6-8716424f4c03 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:48.858397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9000870d-d952-4f2e-a0e6-8716424f4c03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:48.910329 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-9000870d-d952-4f2e-a0e6-8716424f4c03 None None] MFA Rules not processed for user `b03bae15d7bf43bb871c1a2eed6ae09a`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:48.916577 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9000870d-d952-4f2e-a0e6-8716424f4c03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:49.040824 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1090/2176] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:32:48 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:32:49.055074 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73712c8-22cb-44ad-af4a-51cde1292aea admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:32:49.063341 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d73712c8-22cb-44ad-af4a-51cde1292aea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:49.165237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73712c8-22cb-44ad-af4a-51cde1292aea admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:32:49.171516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95aaa20b-d2f4-43df-a5a0-58d33dc0170d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:32:49.173728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95aaa20b-d2f4-43df-a5a0-58d33dc0170d service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:49.174298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95aaa20b-d2f4-43df-a5a0-58d33dc0170d service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:49.174586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95aaa20b-d2f4-43df-a5a0-58d33dc0170d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:49.186418 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95aaa20b-d2f4-43df-a5a0-58d33dc0170d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:32:49.187973 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95aaa20b-d2f4-43df-a5a0-58d33dc0170d service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:32:49.199714 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1087/2177] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:32:49 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:32:50.754414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6184ab85-1a0c-410b-80e1-fd2dbc9f0f60 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:50.754988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6184ab85-1a0c-410b-80e1-fd2dbc9f0f60 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:50.754988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6184ab85-1a0c-410b-80e1-fd2dbc9f0f60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:50.775768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-6184ab85-1a0c-410b-80e1-fd2dbc9f0f60 None None] MFA Rules not processed for user `b03bae15d7bf43bb871c1a2eed6ae09a`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:50.776907 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-6184ab85-1a0c-410b-80e1-fd2dbc9f0f60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:50.784496 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1091/2178] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:32:50 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:32:53.422626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1de0d61-39fc-415d-8f16-b5e052199930 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:53.423198 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1de0d61-39fc-415d-8f16-b5e052199930 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:53.423538 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1de0d61-39fc-415d-8f16-b5e052199930 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:53.424948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1088/2179] 10.222.0.64 () {54 vars in 921 bytes} [Tue Oct 4 14:32:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:32:53.432606 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b064cb25-084d-4b7d-8ab9-2c02fdbdc062 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:32:53.433198 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b064cb25-084d-4b7d-8ab9-2c02fdbdc062 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:32:53.433283 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b064cb25-084d-4b7d-8ab9-2c02fdbdc062 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:32:53.461872 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-b064cb25-084d-4b7d-8ab9-2c02fdbdc062 None None] MFA Rules not processed for user `03081d7d581745c88c12e9d089fcb2d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:32:53.463446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-b064cb25-084d-4b7d-8ab9-2c02fdbdc062 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:32:53.538795 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1092/2180] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Oct 4 14:32:53 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:32:59.194096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=32,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:00.714904 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1de0d61-39fc-415d-8f16-b5e052199930 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:00.717487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c1de0d61-39fc-415d-8f16-b5e052199930 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:00.795463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1de0d61-39fc-415d-8f16-b5e052199930 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:00.803718 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe96e8b-d20e-4fd2-8814-a5241ee0f1c8 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35a5a4ef40204e158649ca25583bbde7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:00.804662 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe96e8b-d20e-4fd2-8814-a5241ee0f1c8 service glance] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:00.807651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe96e8b-d20e-4fd2-8814-a5241ee0f1c8 service glance] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:00.807651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe96e8b-d20e-4fd2-8814-a5241ee0f1c8 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:00.823064 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe96e8b-d20e-4fd2-8814-a5241ee0f1c8 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:00.824559 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe96e8b-d20e-4fd2-8814-a5241ee0f1c8 service glance] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:00.838975 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1089/2181] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:33:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:33:00.942870 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e9c1bd-59a8-4338-834e-05c2a0b301f1 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:00.943746 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e9c1bd-59a8-4338-834e-05c2a0b301f1 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:00.944183 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e9c1bd-59a8-4338-834e-05c2a0b301f1 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:00.946423 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1093/2182] 10.222.0.64 () {54 vars in 921 bytes} [Tue Oct 4 14:33:00 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:33:00.952138 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5e596b-0735-4782-9de4-d8c0fbc5bb56 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:00.954051 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5e596b-0735-4782-9de4-d8c0fbc5bb56 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:00.954051 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5e596b-0735-4782-9de4-d8c0fbc5bb56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:00.984553 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-ea5e596b-0735-4782-9de4-d8c0fbc5bb56 None None] MFA Rules not processed for user `8dbeccfb9a194992959f0b341ef0d4a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:00.986783 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ea5e596b-0735-4782-9de4-d8c0fbc5bb56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:01.063438 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1090/2183] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Oct 4 14:33:00 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:01.113736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e9c1bd-59a8-4338-834e-05c2a0b301f1 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:01.116587 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-52e9c1bd-59a8-4338-834e-05c2a0b301f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:01.138477 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e9c1bd-59a8-4338-834e-05c2a0b301f1 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:01.147235 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81321327-7192-406b-a800-cdb161e35e4d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:01.149675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81321327-7192-406b-a800-cdb161e35e4d service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:01.150236 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81321327-7192-406b-a800-cdb161e35e4d service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:01.150706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81321327-7192-406b-a800-cdb161e35e4d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:01.157669 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:01.162176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81321327-7192-406b-a800-cdb161e35e4d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:01.164473 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81321327-7192-406b-a800-cdb161e35e4d service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:01.177391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1094/2184] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:33:01 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:33:01.366219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95e04a4-7a16-4bd2-819d-09f3d2d26d60 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:01.366509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95e04a4-7a16-4bd2-819d-09f3d2d26d60 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:01.366864 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95e04a4-7a16-4bd2-819d-09f3d2d26d60 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:01.368346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1091/2185] 10.222.0.64 () {54 vars in 921 bytes} [Tue Oct 4 14:33:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:33:01.374670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da49cf58-cdab-45a1-b5ad-2c8a2d338c9e None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:01.375801 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da49cf58-cdab-45a1-b5ad-2c8a2d338c9e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:01.376084 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da49cf58-cdab-45a1-b5ad-2c8a2d338c9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:01.398157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-da49cf58-cdab-45a1-b5ad-2c8a2d338c9e None None] MFA Rules not processed for user `03081d7d581745c88c12e9d089fcb2d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:01.399105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-da49cf58-cdab-45a1-b5ad-2c8a2d338c9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:01.413970 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1095/2186] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Oct 4 14:33:01 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:01.432655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95e04a4-7a16-4bd2-819d-09f3d2d26d60 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:01.462182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95e04a4-7a16-4bd2-819d-09f3d2d26d60 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:01.467343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe63893-b95d-4362-96ec-1e57f6624c35 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:01.467968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe63893-b95d-4362-96ec-1e57f6624c35 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:01.468255 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe63893-b95d-4362-96ec-1e57f6624c35 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:01.468566 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe63893-b95d-4362-96ec-1e57f6624c35 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:01.497088 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe63893-b95d-4362-96ec-1e57f6624c35 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:01.498832 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe63893-b95d-4362-96ec-1e57f6624c35 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:01.540326 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1092/2187] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:33:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:33:06.277378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91aa306b-17cb-4f33-8741-53825037c67f None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:06.278017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91aa306b-17cb-4f33-8741-53825037c67f None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:06.278017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91aa306b-17cb-4f33-8741-53825037c67f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:06.301266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-91aa306b-17cb-4f33-8741-53825037c67f None None] MFA Rules not processed for user `34afa4aac10c4a018ca05a0ba34c940a`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:06.302524 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-91aa306b-17cb-4f33-8741-53825037c67f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:06.395191 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1096/2188] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Oct 4 14:33:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:06.411405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe63893-b95d-4362-96ec-1e57f6624c35 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:06.412871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-abe63893-b95d-4362-96ec-1e57f6624c35 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:06.499010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe63893-b95d-4362-96ec-1e57f6624c35 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:06.505077 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e13d303-6f60-4a24-8c60-0b567ef4d2d0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5980e353164c4873bcdd2af847e0a1f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:06.505685 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e13d303-6f60-4a24-8c60-0b567ef4d2d0 service swift] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:06.505959 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e13d303-6f60-4a24-8c60-0b567ef4d2d0 service swift] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:06.506229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e13d303-6f60-4a24-8c60-0b567ef4d2d0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:06.514891 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e13d303-6f60-4a24-8c60-0b567ef4d2d0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:06.517221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e13d303-6f60-4a24-8c60-0b567ef4d2d0 service swift] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:06.525710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1093/2189] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Oct 4 14:33:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 14:33:13.361197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91aa306b-17cb-4f33-8741-53825037c67f None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:13.362675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-91aa306b-17cb-4f33-8741-53825037c67f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:13.460079 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91aa306b-17cb-4f33-8741-53825037c67f None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:13.465899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2782ab67-5e72-41f1-9320-76b0bd8f5fa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:13.466804 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2782ab67-5e72-41f1-9320-76b0bd8f5fa7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:13.467159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2782ab67-5e72-41f1-9320-76b0bd8f5fa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:13.467497 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2782ab67-5e72-41f1-9320-76b0bd8f5fa7 admin admin] PATH_INFO: `/v3/users/c22d02eea2cd4c9a8aacb54ca70c1ff3` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:13.468960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2782ab67-5e72-41f1-9320-76b0bd8f5fa7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c22d02eea2cd4c9a8aacb54ca70c1ff3)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:13.471234 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2782ab67-5e72-41f1-9320-76b0bd8f5fa7 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:13.549504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2782ab67-5e72-41f1-9320-76b0bd8f5fa7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c22d02eea2cd4c9a8aacb54ca70c1ff3'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:13.550261 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2782ab67-5e72-41f1-9320-76b0bd8f5fa7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c22d02eea2cd4c9a8aacb54ca70c1ff3'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:13.550887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2782ab67-5e72-41f1-9320-76b0bd8f5fa7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c22d02eea2cd4c9a8aacb54ca70c1ff3'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:13.559390 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2782ab67-5e72-41f1-9320-76b0bd8f5fa7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c22d02eea2cd4c9a8aacb54ca70c1ff3'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:13.576270 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2782ab67-5e72-41f1-9320-76b0bd8f5fa7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c22d02eea2cd4c9a8aacb54ca70c1ff3'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:13.591916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1097/2190] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:33:13 2022] DELETE /identity/v3/users/c22d02eea2cd4c9a8aacb54ca70c1ff3 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:33:13.597057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059c3133-e567-4243-9b23-aa90e2357212 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:13.597697 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059c3133-e567-4243-9b23-aa90e2357212 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:13.597697 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059c3133-e567-4243-9b23-aa90e2357212 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:13.641604 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-059c3133-e567-4243-9b23-aa90e2357212 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:13.645843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-059c3133-e567-4243-9b23-aa90e2357212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:13.685039 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:13.722840 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1094/2191] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:33:13.732519 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2782ab67-5e72-41f1-9320-76b0bd8f5fa7 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:13.733965 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-2782ab67-5e72-41f1-9320-76b0bd8f5fa7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:13.748435 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:13.800456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2782ab67-5e72-41f1-9320-76b0bd8f5fa7 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:13.804577 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ebd32a-a847-4b14-a6da-83d08f0255ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:13.805212 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ebd32a-a847-4b14-a6da-83d08f0255ac service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:13.805633 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ebd32a-a847-4b14-a6da-83d08f0255ac service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:13.805917 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ebd32a-a847-4b14-a6da-83d08f0255ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:13.817676 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ebd32a-a847-4b14-a6da-83d08f0255ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:13.820175 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ebd32a-a847-4b14-a6da-83d08f0255ac service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:13.833385 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1098/2192] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:33:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:33:14.181278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059c3133-e567-4243-9b23-aa90e2357212 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:14.182866 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-059c3133-e567-4243-9b23-aa90e2357212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:14.214749 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059c3133-e567-4243-9b23-aa90e2357212 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:14.220321 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea46162-22d9-4f6d-a423-17d7ebfe36b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:14.221034 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea46162-22d9-4f6d-a423-17d7ebfe36b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:14.221314 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea46162-22d9-4f6d-a423-17d7ebfe36b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:14.221583 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea46162-22d9-4f6d-a423-17d7ebfe36b9 admin admin] PATH_INFO: `/v3/projects/62dfe44d9fec40bdb293a1741670fc53` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:14.222720 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea46162-22d9-4f6d-a423-17d7ebfe36b9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=62dfe44d9fec40bdb293a1741670fc53)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:14.224737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea46162-22d9-4f6d-a423-17d7ebfe36b9 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:14.273564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ebd32a-a847-4b14-a6da-83d08f0255ac service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:14.275291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-81ebd32a-a847-4b14-a6da-83d08f0255ac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:14.287435 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-fea46162-22d9-4f6d-a423-17d7ebfe36b9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '62dfe44d9fec40bdb293a1741670fc53', 'request_id': 'req-fea46162-22d9-4f6d-a423-17d7ebfe36b9'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:14.295259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-fea46162-22d9-4f6d-a423-17d7ebfe36b9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '62dfe44d9fec40bdb293a1741670fc53', 'request_id': 'req-fea46162-22d9-4f6d-a423-17d7ebfe36b9'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:14.313713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ebd32a-a847-4b14-a6da-83d08f0255ac service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:14.316716 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-fea46162-22d9-4f6d-a423-17d7ebfe36b9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '62dfe44d9fec40bdb293a1741670fc53', 'request_id': 'req-fea46162-22d9-4f6d-a423-17d7ebfe36b9'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:14.347528 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-fea46162-22d9-4f6d-a423-17d7ebfe36b9 admin admin] The token cache is being invalidate because project 62dfe44d9fec40bdb293a1741670fc53 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:33:14.347742 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-fea46162-22d9-4f6d-a423-17d7ebfe36b9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:14.350260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-fea46162-22d9-4f6d-a423-17d7ebfe36b9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:14.354494 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1095/2193] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:33:14 2022] DELETE /identity/v3/projects/62dfe44d9fec40bdb293a1741670fc53 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:33:14.358062 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2d174e-7a84-45f9-8696-4987cc36584b None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:14.358354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2d174e-7a84-45f9-8696-4987cc36584b None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:14.358635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2d174e-7a84-45f9-8696-4987cc36584b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:14.397526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-6f2d174e-7a84-45f9-8696-4987cc36584b None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:14.398745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6f2d174e-7a84-45f9-8696-4987cc36584b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:14.430633 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e0c9db-e37a-45cd-8d23-57e3b6f41224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:14.431360 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e0c9db-e37a-45cd-8d23-57e3b6f41224 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:14.431858 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e0c9db-e37a-45cd-8d23-57e3b6f41224 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:14.432321 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e0c9db-e37a-45cd-8d23-57e3b6f41224 admin admin] PATH_INFO: `/v3/users/11261fd586e440749ba943fc2255c2a4` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:14.434033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e0c9db-e37a-45cd-8d23-57e3b6f41224 admin admin] RBAC: Authorizing `identity:delete_user(user_id=11261fd586e440749ba943fc2255c2a4)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:14.437172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e0c9db-e37a-45cd-8d23-57e3b6f41224 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:14.514661 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1096/2194] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:14.523133 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-64e0c9db-e37a-45cd-8d23-57e3b6f41224 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '11261fd586e440749ba943fc2255c2a4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:14.523617 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e623b6-5489-434e-8224-00a288171b1c None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:14.523899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e623b6-5489-434e-8224-00a288171b1c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:14.524017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-64e0c9db-e37a-45cd-8d23-57e3b6f41224 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '11261fd586e440749ba943fc2255c2a4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:14.524225 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e623b6-5489-434e-8224-00a288171b1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:14.524710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-64e0c9db-e37a-45cd-8d23-57e3b6f41224 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '11261fd586e440749ba943fc2255c2a4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:14.534232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-64e0c9db-e37a-45cd-8d23-57e3b6f41224 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '11261fd586e440749ba943fc2255c2a4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:14.552256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-64e0c9db-e37a-45cd-8d23-57e3b6f41224 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '11261fd586e440749ba943fc2255c2a4'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:14.561838 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-f3e623b6-5489-434e-8224-00a288171b1c None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:14.563420 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f3e623b6-5489-434e-8224-00a288171b1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:14.567953 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1099/2195] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:33:14 2022] DELETE /identity/v3/users/11261fd586e440749ba943fc2255c2a4 => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:33:14.571784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1097/2196] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:33:14.572412 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560ca64d-fcc1-483b-a66a-bb66a61e4d07 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:14.572684 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560ca64d-fcc1-483b-a66a-bb66a61e4d07 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:14.572965 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560ca64d-fcc1-483b-a66a-bb66a61e4d07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:14.576052 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e623b6-5489-434e-8224-00a288171b1c None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:14.585391 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e623b6-5489-434e-8224-00a288171b1c None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:14.593411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-560ca64d-fcc1-483b-a66a-bb66a61e4d07 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:14.595380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-560ca64d-fcc1-483b-a66a-bb66a61e4d07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:14.629998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:14.660333 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:14.671848 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1100/2197] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:14.678394 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9685e9-7f7a-4ba2-bc43-3dc4f4b306a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:14.679014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9685e9-7f7a-4ba2-bc43-3dc4f4b306a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:14.679295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9685e9-7f7a-4ba2-bc43-3dc4f4b306a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:14.679565 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9685e9-7f7a-4ba2-bc43-3dc4f4b306a1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:14.680224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9685e9-7f7a-4ba2-bc43-3dc4f4b306a1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:33:14.680713 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9685e9-7f7a-4ba2-bc43-3dc4f4b306a1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:14.682323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9685e9-7f7a-4ba2-bc43-3dc4f4b306a1 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:14.682841 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560ca64d-fcc1-483b-a66a-bb66a61e4d07 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:14.684244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-560ca64d-fcc1-483b-a66a-bb66a61e4d07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:14.695796 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1098/2198] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:33:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:33:14.700128 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9685e9-7f7a-4ba2-bc43-3dc4f4b306a1 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:14.712835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9685e9-7f7a-4ba2-bc43-3dc4f4b306a1 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:14.719771 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df164a3-5a8f-4431-a818-7194d8788d1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:14.720417 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df164a3-5a8f-4431-a818-7194d8788d1d admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:14.720828 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df164a3-5a8f-4431-a818-7194d8788d1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:14.721106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df164a3-5a8f-4431-a818-7194d8788d1d admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:14.721749 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df164a3-5a8f-4431-a818-7194d8788d1d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:14.723672 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df164a3-5a8f-4431-a818-7194d8788d1d admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:14.761085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1099/2199] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:33:14 2022] POST /identity/v3/projects => generated 371 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:33:14.762574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df164a3-5a8f-4431-a818-7194d8788d1d admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:14.779130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560ca64d-fcc1-483b-a66a-bb66a61e4d07 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:14.857054 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df164a3-5a8f-4431-a818-7194d8788d1d admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:14.863018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8066bbc1-c330-4279-a491-0f7b20a26724 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:14.863640 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8066bbc1-c330-4279-a491-0f7b20a26724 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:14.863919 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8066bbc1-c330-4279-a491-0f7b20a26724 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:14.864196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8066bbc1-c330-4279-a491-0f7b20a26724 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:14.864873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8066bbc1-c330-4279-a491-0f7b20a26724 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:14.866822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8066bbc1-c330-4279-a491-0f7b20a26724 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:14.890100 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d78678b-a58f-41e6-9c02-8a353e2cf828 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:14.891776 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d78678b-a58f-41e6-9c02-8a353e2cf828 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:14.892868 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d78678b-a58f-41e6-9c02-8a353e2cf828 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:14.893149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d78678b-a58f-41e6-9c02-8a353e2cf828 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:14.905353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1100/2200] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:33:14 2022] POST /identity/v3/users => generated 286 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:33:14.906316 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8066bbc1-c330-4279-a491-0f7b20a26724 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:14.916636 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d78678b-a58f-41e6-9c02-8a353e2cf828 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:14.918640 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d78678b-a58f-41e6-9c02-8a353e2cf828 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:14.922857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8066bbc1-c330-4279-a491-0f7b20a26724 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:14.931986 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1101/2201] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:33:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:33:14.932200 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f6e769-c4b3-4cdf-80bf-97b56acff074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:14.933435 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f6e769-c4b3-4cdf-80bf-97b56acff074 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:14.933784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f6e769-c4b3-4cdf-80bf-97b56acff074 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:14.934058 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f6e769-c4b3-4cdf-80bf-97b56acff074 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:14.934738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f6e769-c4b3-4cdf-80bf-97b56acff074 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:14.936356 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f6e769-c4b3-4cdf-80bf-97b56acff074 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:14.947772 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1101/2202] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:33:14 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:33:14.962151 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d78678b-a58f-41e6-9c02-8a353e2cf828 service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:14.975337 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d78678b-a58f-41e6-9c02-8a353e2cf828 service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:14.980835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ca91a5-1d21-4062-8cf0-4e0524cb7fba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:14.996619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ca91a5-1d21-4062-8cf0-4e0524cb7fba admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:14.996619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ca91a5-1d21-4062-8cf0-4e0524cb7fba admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:14.996619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ca91a5-1d21-4062-8cf0-4e0524cb7fba admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:14.996619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ca91a5-1d21-4062-8cf0-4e0524cb7fba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:14.996619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ca91a5-1d21-4062-8cf0-4e0524cb7fba admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:15.002010 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1102/2203] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:33:14 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:33:15.003219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f6e769-c4b3-4cdf-80bf-97b56acff074 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:15.036052 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f6e769-c4b3-4cdf-80bf-97b56acff074 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:15.044511 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ece72e-131c-465e-b2b5-73ff022258aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:15.045660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ece72e-131c-465e-b2b5-73ff022258aa admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:15.046314 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ece72e-131c-465e-b2b5-73ff022258aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:15.046708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ece72e-131c-465e-b2b5-73ff022258aa admin admin] PATH_INFO: `/v3/projects/e8973f9721be41059fc160143df3951e/users/279c7bf11b7a41c4b3abe0f3e3d4dc44/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:15.099290 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ece72e-131c-465e-b2b5-73ff022258aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8973f9721be41059fc160143df3951e, user_id=279c7bf11b7a41c4b3abe0f3e3d4dc44, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:15.104580 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ece72e-131c-465e-b2b5-73ff022258aa admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:15.124184 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1102/2204] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:33:15 2022] PUT /identity/v3/projects/e8973f9721be41059fc160143df3951e/users/279c7bf11b7a41c4b3abe0f3e3d4dc44/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:33:15.134977 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a614a38-7c6b-4a8d-90dc-80aa35259ba0 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:15.135264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a614a38-7c6b-4a8d-90dc-80aa35259ba0 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:15.135546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a614a38-7c6b-4a8d-90dc-80aa35259ba0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:15.182671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-6a614a38-7c6b-4a8d-90dc-80aa35259ba0 None None] MFA Rules not processed for user `279c7bf11b7a41c4b3abe0f3e3d4dc44`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:15.189418 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-6a614a38-7c6b-4a8d-90dc-80aa35259ba0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:15.300353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1103/2205] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:15 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:15.313799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ece72e-131c-465e-b2b5-73ff022258aa admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:15.316392 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f0ece72e-131c-465e-b2b5-73ff022258aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:15.384118 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a614a38-7c6b-4a8d-90dc-80aa35259ba0 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:15.386138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-6a614a38-7c6b-4a8d-90dc-80aa35259ba0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:15.401113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ece72e-131c-465e-b2b5-73ff022258aa admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:15.405312 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:15.408399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a36c3a-d154-4340-9cc4-2ade88875bcb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:15.409237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a36c3a-d154-4340-9cc4-2ade88875bcb service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:15.409721 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a36c3a-d154-4340-9cc4-2ade88875bcb service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:15.410180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a36c3a-d154-4340-9cc4-2ade88875bcb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:15.420879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a36c3a-d154-4340-9cc4-2ade88875bcb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:15.422410 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a36c3a-d154-4340-9cc4-2ade88875bcb service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:15.432891 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1103/2206] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:33:15 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:33:15.463976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a36c3a-d154-4340-9cc4-2ade88875bcb service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:15.492792 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:15.506562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a614a38-7c6b-4a8d-90dc-80aa35259ba0 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:15.518055 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf7229d-3061-4a49-8292-3879933e9824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:15.519086 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf7229d-3061-4a49-8292-3879933e9824 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:15.519878 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf7229d-3061-4a49-8292-3879933e9824 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:15.520933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf7229d-3061-4a49-8292-3879933e9824 admin admin] PATH_INFO: `/v3/projects/c9fc3a9be06c446db32b810b6a614dd5` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:15.523177 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf7229d-3061-4a49-8292-3879933e9824 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c9fc3a9be06c446db32b810b6a614dd5)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:15.526778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf7229d-3061-4a49-8292-3879933e9824 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:15.590590 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a36c3a-d154-4340-9cc4-2ade88875bcb service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:15.618902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2bf7229d-3061-4a49-8292-3879933e9824 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c9fc3a9be06c446db32b810b6a614dd5', 'request_id': 'req-2bf7229d-3061-4a49-8292-3879933e9824'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:15.633300 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2bf7229d-3061-4a49-8292-3879933e9824 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c9fc3a9be06c446db32b810b6a614dd5', 'request_id': 'req-2bf7229d-3061-4a49-8292-3879933e9824'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:15.642563 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2bf7229d-3061-4a49-8292-3879933e9824 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c9fc3a9be06c446db32b810b6a614dd5', 'request_id': 'req-2bf7229d-3061-4a49-8292-3879933e9824'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:15.671341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2bf7229d-3061-4a49-8292-3879933e9824 admin admin] The token cache is being invalidate because project c9fc3a9be06c446db32b810b6a614dd5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:33:15.671711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2bf7229d-3061-4a49-8292-3879933e9824 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:15.674464 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2bf7229d-3061-4a49-8292-3879933e9824 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:15.676039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1104/2207] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:33:15 2022] DELETE /identity/v3/projects/c9fc3a9be06c446db32b810b6a614dd5 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:33:15.680892 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b9faef-e6d3-4ca9-bd91-c2184b2b3dcc None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:15.681173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b9faef-e6d3-4ca9-bd91-c2184b2b3dcc None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:15.681449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b9faef-e6d3-4ca9-bd91-c2184b2b3dcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:15.695492 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e070e683-3e5e-46de-a0ff-5e889bbafc80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:15.696277 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e070e683-3e5e-46de-a0ff-5e889bbafc80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:15.696663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e070e683-3e5e-46de-a0ff-5e889bbafc80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:15.696941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e070e683-3e5e-46de-a0ff-5e889bbafc80 admin admin] PATH_INFO: `/v3/users/279c7bf11b7a41c4b3abe0f3e3d4dc44` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:15.698136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e070e683-3e5e-46de-a0ff-5e889bbafc80 admin admin] RBAC: Authorizing `identity:delete_user(user_id=279c7bf11b7a41c4b3abe0f3e3d4dc44)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:15.700152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e070e683-3e5e-46de-a0ff-5e889bbafc80 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:15.703115 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-06b9faef-e6d3-4ca9-bd91-c2184b2b3dcc None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:15.704790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-06b9faef-e6d3-4ca9-bd91-c2184b2b3dcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:15.713942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1105/2208] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:15.722739 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccc4d12-0716-497f-b17d-5dd6547e3998 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:15.723037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccc4d12-0716-497f-b17d-5dd6547e3998 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:15.723440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccc4d12-0716-497f-b17d-5dd6547e3998 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:15.752177 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-fccc4d12-0716-497f-b17d-5dd6547e3998 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:15.753653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-fccc4d12-0716-497f-b17d-5dd6547e3998 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:15.772634 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e070e683-3e5e-46de-a0ff-5e889bbafc80 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '279c7bf11b7a41c4b3abe0f3e3d4dc44'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:15.785298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e070e683-3e5e-46de-a0ff-5e889bbafc80 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '279c7bf11b7a41c4b3abe0f3e3d4dc44'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:15.797034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1106/2209] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:15.801568 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fccc4d12-0716-497f-b17d-5dd6547e3998 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:15.802169 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e070e683-3e5e-46de-a0ff-5e889bbafc80 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '279c7bf11b7a41c4b3abe0f3e3d4dc44'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:15.817012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fccc4d12-0716-497f-b17d-5dd6547e3998 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:15.823251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c67cda-4afe-499d-bb8a-009b154a5d6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:15.823949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c67cda-4afe-499d-bb8a-009b154a5d6b admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:15.824225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c67cda-4afe-499d-bb8a-009b154a5d6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:15.824735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c67cda-4afe-499d-bb8a-009b154a5d6b admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:15.825534 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c67cda-4afe-499d-bb8a-009b154a5d6b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:33:15.825836 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c67cda-4afe-499d-bb8a-009b154a5d6b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:15.827791 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c67cda-4afe-499d-bb8a-009b154a5d6b admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:15.832348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e070e683-3e5e-46de-a0ff-5e889bbafc80 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '279c7bf11b7a41c4b3abe0f3e3d4dc44'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:15.833117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e070e683-3e5e-46de-a0ff-5e889bbafc80 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '279c7bf11b7a41c4b3abe0f3e3d4dc44'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:15.856248 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1107/2211] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:33:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:33:15.856680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1104/2211] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:33:15 2022] DELETE /identity/v3/users/279c7bf11b7a41c4b3abe0f3e3d4dc44 => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:33:15.857533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c67cda-4afe-499d-bb8a-009b154a5d6b admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:15.858559 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c9ac1c-4518-4253-80c2-47a61d0c8001 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:15.859105 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c9ac1c-4518-4253-80c2-47a61d0c8001 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:15.860342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c9ac1c-4518-4253-80c2-47a61d0c8001 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:15.861916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-31c67cda-4afe-499d-bb8a-009b154a5d6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:15.885664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-b3c9ac1c-4518-4253-80c2-47a61d0c8001 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:15.887289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b3c9ac1c-4518-4253-80c2-47a61d0c8001 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:15.887820 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c67cda-4afe-499d-bb8a-009b154a5d6b admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:15.950328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:15.965376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cb4b25-80d0-46bd-9998-fd11ead4b166 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:15.965972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cb4b25-80d0-46bd-9998-fd11ead4b166 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:15.966256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cb4b25-80d0-46bd-9998-fd11ead4b166 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:15.966527 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cb4b25-80d0-46bd-9998-fd11ead4b166 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:15.967159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67cb4b25-80d0-46bd-9998-fd11ead4b166 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:15.969173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67cb4b25-80d0-46bd-9998-fd11ead4b166 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:15.975953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1105/2212] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:15.989062 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c9ac1c-4518-4253-80c2-47a61d0c8001 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:15.990899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b3c9ac1c-4518-4253-80c2-47a61d0c8001 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:16.018232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:16.034721 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1108/2213] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:33:15 2022] POST /identity/v3/projects => generated 391 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:33:16.039727 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cb4b25-80d0-46bd-9998-fd11ead4b166 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:16.104525 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c9ac1c-4518-4253-80c2-47a61d0c8001 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:16.170813 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cb4b25-80d0-46bd-9998-fd11ead4b166 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:16.179535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a3a36e-17fc-4860-8e92-8600436e5db5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:16.180475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a3a36e-17fc-4860-8e92-8600436e5db5 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:16.180660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a3a36e-17fc-4860-8e92-8600436e5db5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:16.180930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a3a36e-17fc-4860-8e92-8600436e5db5 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:16.181570 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a3a36e-17fc-4860-8e92-8600436e5db5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:16.183539 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a3a36e-17fc-4860-8e92-8600436e5db5 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:16.201542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74d40be-9038-4a39-ac3b-36482a368d19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:16.202171 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74d40be-9038-4a39-ac3b-36482a368d19 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:16.202524 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74d40be-9038-4a39-ac3b-36482a368d19 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:16.203008 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74d40be-9038-4a39-ac3b-36482a368d19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:16.214887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1109/2214] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:33:16 2022] POST /identity/v3/users => generated 296 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:33:16.218793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a3a36e-17fc-4860-8e92-8600436e5db5 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:16.219458 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74d40be-9038-4a39-ac3b-36482a368d19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:16.221267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74d40be-9038-4a39-ac3b-36482a368d19 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:16.233313 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a3a36e-17fc-4860-8e92-8600436e5db5 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:16.235994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1106/2215] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:33:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:33:16.239671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bb5ee3-4d72-490c-b535-42f8f00df803 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:16.240302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bb5ee3-4d72-490c-b535-42f8f00df803 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:16.240608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bb5ee3-4d72-490c-b535-42f8f00df803 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:16.240878 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bb5ee3-4d72-490c-b535-42f8f00df803 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:16.241518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bb5ee3-4d72-490c-b535-42f8f00df803 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:16.243529 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bb5ee3-4d72-490c-b535-42f8f00df803 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:16.257933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1110/2216] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:33:16 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:33:16.259459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74d40be-9038-4a39-ac3b-36482a368d19 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:16.271278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74d40be-9038-4a39-ac3b-36482a368d19 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:16.276757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953a9dfe-b18f-46cd-9ba8-b3ef1fee91ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:16.277376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953a9dfe-b18f-46cd-9ba8-b3ef1fee91ad admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:16.277667 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953a9dfe-b18f-46cd-9ba8-b3ef1fee91ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:16.277985 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953a9dfe-b18f-46cd-9ba8-b3ef1fee91ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:16.278641 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953a9dfe-b18f-46cd-9ba8-b3ef1fee91ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:16.280247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953a9dfe-b18f-46cd-9ba8-b3ef1fee91ad admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:16.294896 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1107/2217] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:33:16 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:33:16.296219 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bb5ee3-4d72-490c-b535-42f8f00df803 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:16.309977 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bb5ee3-4d72-490c-b535-42f8f00df803 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:16.317504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062ee58f-6442-456e-817c-60dc52e3ff1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:16.319272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062ee58f-6442-456e-817c-60dc52e3ff1c admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:16.320028 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062ee58f-6442-456e-817c-60dc52e3ff1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:16.320887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062ee58f-6442-456e-817c-60dc52e3ff1c admin admin] PATH_INFO: `/v3/projects/cfeec3896145481bbace81ebf6916795/users/c919ce94155f4c049f263dec7588562d/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:16.341923 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062ee58f-6442-456e-817c-60dc52e3ff1c admin admin] RBAC: Authorizing `identity:create_grant(project_id=cfeec3896145481bbace81ebf6916795, user_id=c919ce94155f4c049f263dec7588562d, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:16.345225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062ee58f-6442-456e-817c-60dc52e3ff1c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:16.363450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1111/2218] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:33:16 2022] PUT /identity/v3/projects/cfeec3896145481bbace81ebf6916795/users/c919ce94155f4c049f263dec7588562d/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:33:16.374387 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b810609-957b-4b40-b317-8fef9946984b None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:16.374675 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b810609-957b-4b40-b317-8fef9946984b None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:16.374970 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b810609-957b-4b40-b317-8fef9946984b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:16.432836 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-1b810609-957b-4b40-b317-8fef9946984b None None] MFA Rules not processed for user `c919ce94155f4c049f263dec7588562d`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:16.434066 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1b810609-957b-4b40-b317-8fef9946984b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:16.586262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062ee58f-6442-456e-817c-60dc52e3ff1c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:16.588556 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-062ee58f-6442-456e-817c-60dc52e3ff1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:16.589095 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1108/2219] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:16 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:16.594794 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b810609-957b-4b40-b317-8fef9946984b None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:16.599135 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1b810609-957b-4b40-b317-8fef9946984b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:16.630807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:16.675387 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b810609-957b-4b40-b317-8fef9946984b None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:16.680503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b834c3-e589-4931-989f-5e1dbaed2fb2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:16.681255 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b834c3-e589-4931-989f-5e1dbaed2fb2 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:16.681641 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b834c3-e589-4931-989f-5e1dbaed2fb2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:16.682019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b834c3-e589-4931-989f-5e1dbaed2fb2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:16.693859 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b834c3-e589-4931-989f-5e1dbaed2fb2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:16.695492 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b834c3-e589-4931-989f-5e1dbaed2fb2 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:16.706557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1109/2220] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:33:16 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:33:16.731732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062ee58f-6442-456e-817c-60dc52e3ff1c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:16.737550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254a71a2-a3f7-40a4-9120-ff0a48d0219e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:16.738406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254a71a2-a3f7-40a4-9120-ff0a48d0219e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:16.738856 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254a71a2-a3f7-40a4-9120-ff0a48d0219e admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:16.739289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254a71a2-a3f7-40a4-9120-ff0a48d0219e admin admin] PATH_INFO: `/v3/projects/e8973f9721be41059fc160143df3951e` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:16.740731 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254a71a2-a3f7-40a4-9120-ff0a48d0219e admin admin] RBAC: Authorizing `identity:delete_project(project_id=e8973f9721be41059fc160143df3951e)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:16.742860 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254a71a2-a3f7-40a4-9120-ff0a48d0219e admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:16.825004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-254a71a2-a3f7-40a4-9120-ff0a48d0219e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e8973f9721be41059fc160143df3951e', 'request_id': 'req-254a71a2-a3f7-40a4-9120-ff0a48d0219e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:16.841480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-254a71a2-a3f7-40a4-9120-ff0a48d0219e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e8973f9721be41059fc160143df3951e', 'request_id': 'req-254a71a2-a3f7-40a4-9120-ff0a48d0219e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:16.844684 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=91,keystone:INSERT=4,keystone:DELETE=5 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:16.850559 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-254a71a2-a3f7-40a4-9120-ff0a48d0219e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e8973f9721be41059fc160143df3951e', 'request_id': 'req-254a71a2-a3f7-40a4-9120-ff0a48d0219e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:16.894209 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-254a71a2-a3f7-40a4-9120-ff0a48d0219e admin admin] The token cache is being invalidate because project e8973f9721be41059fc160143df3951e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:33:16.894579 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-254a71a2-a3f7-40a4-9120-ff0a48d0219e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:16.895340 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-254a71a2-a3f7-40a4-9120-ff0a48d0219e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:16.920347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1112/2221] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:33:16 2022] DELETE /identity/v3/projects/e8973f9721be41059fc160143df3951e => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:33:26.701009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=17 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:26.872200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=4,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:36.698740 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b834c3-e589-4931-989f-5e1dbaed2fb2 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:36.713582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b834c3-e589-4931-989f-5e1dbaed2fb2 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:36.791340 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ddfd42-9a43-42de-86e1-24b443e59b6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:36.791974 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ddfd42-9a43-42de-86e1-24b443e59b6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:36.792258 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ddfd42-9a43-42de-86e1-24b443e59b6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:36.792563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ddfd42-9a43-42de-86e1-24b443e59b6b admin admin] PATH_INFO: `/v3/users/c919ce94155f4c049f263dec7588562d` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:36.793710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ddfd42-9a43-42de-86e1-24b443e59b6b admin admin] RBAC: Authorizing `identity:delete_user(user_id=c919ce94155f4c049f263dec7588562d)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:36.796992 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ddfd42-9a43-42de-86e1-24b443e59b6b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:36.883915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-63ddfd42-9a43-42de-86e1-24b443e59b6b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c919ce94155f4c049f263dec7588562d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:36.895826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-63ddfd42-9a43-42de-86e1-24b443e59b6b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c919ce94155f4c049f263dec7588562d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:36.905980 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-63ddfd42-9a43-42de-86e1-24b443e59b6b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c919ce94155f4c049f263dec7588562d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:36.921421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-63ddfd42-9a43-42de-86e1-24b443e59b6b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c919ce94155f4c049f263dec7588562d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:36.923863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-63ddfd42-9a43-42de-86e1-24b443e59b6b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c919ce94155f4c049f263dec7588562d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:36.930100 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1110/2222] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:33:36 2022] DELETE /identity/v3/users/c919ce94155f4c049f263dec7588562d => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:33:36.936126 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8a790e-0e45-40b4-b925-ab1f5ca69b81 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:36.936679 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8a790e-0e45-40b4-b925-ab1f5ca69b81 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:36.936766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8a790e-0e45-40b4-b925-ab1f5ca69b81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:36.966146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-3b8a790e-0e45-40b4-b925-ab1f5ca69b81 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:36.967382 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-3b8a790e-0e45-40b4-b925-ab1f5ca69b81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:37.056489 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1113/2223] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:37.070257 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ddfd42-9a43-42de-86e1-24b443e59b6b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:37.072645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-63ddfd42-9a43-42de-86e1-24b443e59b6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:37.144940 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ddfd42-9a43-42de-86e1-24b443e59b6b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:37.149912 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd71f872-e779-4599-acab-07aa399f77f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:37.150535 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd71f872-e779-4599-acab-07aa399f77f8 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:37.150813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd71f872-e779-4599-acab-07aa399f77f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:37.151081 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd71f872-e779-4599-acab-07aa399f77f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:37.167481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd71f872-e779-4599-acab-07aa399f77f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:37.168983 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd71f872-e779-4599-acab-07aa399f77f8 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:37.181298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1111/2224] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:33:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:33:37.627464 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8a790e-0e45-40b4-b925-ab1f5ca69b81 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:37.628997 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-3b8a790e-0e45-40b4-b925-ab1f5ca69b81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:37.642257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8a790e-0e45-40b4-b925-ab1f5ca69b81 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:37.648596 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952b32be-f152-4863-8b82-0ed6e8c0e4c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:37.648596 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952b32be-f152-4863-8b82-0ed6e8c0e4c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:37.649024 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952b32be-f152-4863-8b82-0ed6e8c0e4c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:37.649024 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952b32be-f152-4863-8b82-0ed6e8c0e4c6 admin admin] PATH_INFO: `/v3/projects/cfeec3896145481bbace81ebf6916795` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:37.664798 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952b32be-f152-4863-8b82-0ed6e8c0e4c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cfeec3896145481bbace81ebf6916795)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:37.667181 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952b32be-f152-4863-8b82-0ed6e8c0e4c6 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:37.748742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-952b32be-f152-4863-8b82-0ed6e8c0e4c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cfeec3896145481bbace81ebf6916795', 'request_id': 'req-952b32be-f152-4863-8b82-0ed6e8c0e4c6'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:37.758036 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-952b32be-f152-4863-8b82-0ed6e8c0e4c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cfeec3896145481bbace81ebf6916795', 'request_id': 'req-952b32be-f152-4863-8b82-0ed6e8c0e4c6'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:37.769272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-952b32be-f152-4863-8b82-0ed6e8c0e4c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cfeec3896145481bbace81ebf6916795', 'request_id': 'req-952b32be-f152-4863-8b82-0ed6e8c0e4c6'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:37.789330 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-952b32be-f152-4863-8b82-0ed6e8c0e4c6 admin admin] The token cache is being invalidate because project cfeec3896145481bbace81ebf6916795 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:33:37.789704 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-952b32be-f152-4863-8b82-0ed6e8c0e4c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:37.790940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-952b32be-f152-4863-8b82-0ed6e8c0e4c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:37.792955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1114/2225] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:33:37 2022] DELETE /identity/v3/projects/cfeec3896145481bbace81ebf6916795 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:33:37.798586 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab08707-6cdc-4132-8db7-380421a716ad None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:37.799406 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab08707-6cdc-4132-8db7-380421a716ad None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:37.799406 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab08707-6cdc-4132-8db7-380421a716ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:37.829632 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-8ab08707-6cdc-4132-8db7-380421a716ad None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:37.831126 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-8ab08707-6cdc-4132-8db7-380421a716ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:37.854738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:37.917769 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1112/2226] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:33:37.927548 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c513b49-3006-443f-bd27-2ad6b6ebee6a None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:37.927998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c513b49-3006-443f-bd27-2ad6b6ebee6a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:37.928484 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c513b49-3006-443f-bd27-2ad6b6ebee6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:37.963229 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-0c513b49-3006-443f-bd27-2ad6b6ebee6a None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:37.965384 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0c513b49-3006-443f-bd27-2ad6b6ebee6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:37.977127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1115/2227] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:37.982238 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab08707-6cdc-4132-8db7-380421a716ad None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:37.994215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab08707-6cdc-4132-8db7-380421a716ad None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:37.999620 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5188616f-de83-4ce9-a3fc-390f517bd363 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:38.000252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5188616f-de83-4ce9-a3fc-390f517bd363 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:38.000565 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5188616f-de83-4ce9-a3fc-390f517bd363 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:38.001441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5188616f-de83-4ce9-a3fc-390f517bd363 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:38.002188 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5188616f-de83-4ce9-a3fc-390f517bd363 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:33:38.002682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5188616f-de83-4ce9-a3fc-390f517bd363 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:38.004634 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5188616f-de83-4ce9-a3fc-390f517bd363 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:38.020290 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1113/2228] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:33:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:33:38.024463 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c513b49-3006-443f-bd27-2ad6b6ebee6a None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:38.039023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c513b49-3006-443f-bd27-2ad6b6ebee6a None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:38.045801 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff49fc03-5635-4185-a8e8-d008a513a46a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:38.046490 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff49fc03-5635-4185-a8e8-d008a513a46a admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:38.046911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff49fc03-5635-4185-a8e8-d008a513a46a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:38.047259 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff49fc03-5635-4185-a8e8-d008a513a46a admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:38.049053 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff49fc03-5635-4185-a8e8-d008a513a46a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:38.051806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff49fc03-5635-4185-a8e8-d008a513a46a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:38.090214 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1116/2229] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:33:38 2022] POST /identity/v3/projects => generated 383 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:33:38.098679 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5188616f-de83-4ce9-a3fc-390f517bd363 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:38.109956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5188616f-de83-4ce9-a3fc-390f517bd363 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:38.179687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6129fc-5fdc-455f-9cc8-903f3fbe3a4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:38.180296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6129fc-5fdc-455f-9cc8-903f3fbe3a4b admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:38.180597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6129fc-5fdc-455f-9cc8-903f3fbe3a4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:38.180864 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6129fc-5fdc-455f-9cc8-903f3fbe3a4b admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:38.181525 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6129fc-5fdc-455f-9cc8-903f3fbe3a4b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:38.183458 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6129fc-5fdc-455f-9cc8-903f3fbe3a4b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:38.216699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1114/2230] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:33:38 2022] POST /identity/v3/users => generated 292 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:33:38.223573 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff49fc03-5635-4185-a8e8-d008a513a46a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:38.256454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff49fc03-5635-4185-a8e8-d008a513a46a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:38.264808 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05677c7-542b-4297-95bf-24993182999c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:38.267071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05677c7-542b-4297-95bf-24993182999c admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:38.267841 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05677c7-542b-4297-95bf-24993182999c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:38.268613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05677c7-542b-4297-95bf-24993182999c admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:38.269660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05677c7-542b-4297-95bf-24993182999c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:38.272228 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05677c7-542b-4297-95bf-24993182999c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:38.293680 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1117/2231] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:33:38 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:33:38.299083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6129fc-5fdc-455f-9cc8-903f3fbe3a4b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:38.314845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6129fc-5fdc-455f-9cc8-903f3fbe3a4b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:38.320054 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3637c510-317d-4cb9-939f-2eff008c9531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:38.322021 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3637c510-317d-4cb9-939f-2eff008c9531 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:38.322299 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3637c510-317d-4cb9-939f-2eff008c9531 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:38.322570 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3637c510-317d-4cb9-939f-2eff008c9531 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:38.323238 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3637c510-317d-4cb9-939f-2eff008c9531 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:38.325044 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3637c510-317d-4cb9-939f-2eff008c9531 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:38.350651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1115/2232] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:33:38 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:33:38.351399 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05677c7-542b-4297-95bf-24993182999c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:38.367464 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05677c7-542b-4297-95bf-24993182999c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:38.373444 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4522621f-a93a-4c37-962c-6102e0ced33c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:38.374210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4522621f-a93a-4c37-962c-6102e0ced33c admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:38.374495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4522621f-a93a-4c37-962c-6102e0ced33c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:38.374786 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4522621f-a93a-4c37-962c-6102e0ced33c admin admin] PATH_INFO: `/v3/projects/9cf9a725cf7e42ea939bee84113b12ee/users/cab40292db0d4a74b1fd59c3539e81e3/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:38.399514 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4522621f-a93a-4c37-962c-6102e0ced33c admin admin] RBAC: Authorizing `identity:create_grant(project_id=9cf9a725cf7e42ea939bee84113b12ee, user_id=cab40292db0d4a74b1fd59c3539e81e3, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:38.404080 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4522621f-a93a-4c37-962c-6102e0ced33c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:38.429020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1118/2233] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:33:38 2022] PUT /identity/v3/projects/9cf9a725cf7e42ea939bee84113b12ee/users/cab40292db0d4a74b1fd59c3539e81e3/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:33:38.434775 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf63664-5d35-4fd9-aa3c-97503fdb2d03 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:38.435206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf63664-5d35-4fd9-aa3c-97503fdb2d03 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:38.435577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf63664-5d35-4fd9-aa3c-97503fdb2d03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:38.461379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-4cf63664-5d35-4fd9-aa3c-97503fdb2d03 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:38.462909 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-4cf63664-5d35-4fd9-aa3c-97503fdb2d03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:38.538057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1116/2234] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:38.560463 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4522621f-a93a-4c37-962c-6102e0ced33c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:38.562313 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-4522621f-a93a-4c37-962c-6102e0ced33c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:38.594180 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:38.641079 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354b8444-556f-4331-b3e4-33decff26b22 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:38.642193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354b8444-556f-4331-b3e4-33decff26b22 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:38.642696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354b8444-556f-4331-b3e4-33decff26b22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:38.651223 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4522621f-a93a-4c37-962c-6102e0ced33c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:38.655171 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b28e02-3771-43ee-af66-bcf2478fd33d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:38.655803 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b28e02-3771-43ee-af66-bcf2478fd33d service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:38.656110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b28e02-3771-43ee-af66-bcf2478fd33d service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:38.656850 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b28e02-3771-43ee-af66-bcf2478fd33d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:38.668077 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b28e02-3771-43ee-af66-bcf2478fd33d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:38.669558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b28e02-3771-43ee-af66-bcf2478fd33d service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:38.670974 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-354b8444-556f-4331-b3e4-33decff26b22 None None] MFA Rules not processed for user `34afa4aac10c4a018ca05a0ba34c940a`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:38.672199 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-354b8444-556f-4331-b3e4-33decff26b22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:38.681147 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1119/2235] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:33:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:33:38.707776 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:38.760729 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1117/2236] 10.222.0.64 () {58 vars in 967 bytes} [Tue Oct 4 14:33:38 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:33:38.794913 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b28e02-3771-43ee-af66-bcf2478fd33d service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:38.797533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d1b28e02-3771-43ee-af66-bcf2478fd33d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:38.909118 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b28e02-3771-43ee-af66-bcf2478fd33d service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:38.919023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bc3317-4a3b-45dd-8f37-f86eb3ca233d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5980e353164c4873bcdd2af847e0a1f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:38.920265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bc3317-4a3b-45dd-8f37-f86eb3ca233d service swift] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:38.920917 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bc3317-4a3b-45dd-8f37-f86eb3ca233d service swift] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:38.921446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bc3317-4a3b-45dd-8f37-f86eb3ca233d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:38.933500 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bc3317-4a3b-45dd-8f37-f86eb3ca233d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:38.935215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bc3317-4a3b-45dd-8f37-f86eb3ca233d service swift] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:38.946133 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1120/2237] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Oct 4 14:33:38 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 14:33:42.462260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372fa057-b70e-42ca-b96e-583dc0f2ef8c None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:42.462917 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372fa057-b70e-42ca-b96e-583dc0f2ef8c None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:42.463060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372fa057-b70e-42ca-b96e-583dc0f2ef8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:42.504700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-372fa057-b70e-42ca-b96e-583dc0f2ef8c None None] MFA Rules not processed for user `34afa4aac10c4a018ca05a0ba34c940a`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:42.506722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-372fa057-b70e-42ca-b96e-583dc0f2ef8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:42.517515 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1118/2238] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Oct 4 14:33:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:42.538995 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bc3317-4a3b-45dd-8f37-f86eb3ca233d service swift] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:42.562581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bc3317-4a3b-45dd-8f37-f86eb3ca233d service swift] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:42.584008 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4ce107-3ae6-4641-8d1b-ea3e5a140e53 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5980e353164c4873bcdd2af847e0a1f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:42.585145 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4ce107-3ae6-4641-8d1b-ea3e5a140e53 service swift] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:42.585590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4ce107-3ae6-4641-8d1b-ea3e5a140e53 service swift] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:42.596588 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4ce107-3ae6-4641-8d1b-ea3e5a140e53 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:42.631090 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4ce107-3ae6-4641-8d1b-ea3e5a140e53 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:42.632968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4ce107-3ae6-4641-8d1b-ea3e5a140e53 service swift] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:42.653727 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1121/2239] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Oct 4 14:33:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 14:33:43.539109 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f334cc3-b1a6-48cb-8e9b-0633e8fecb0a None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:43.539672 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f334cc3-b1a6-48cb-8e9b-0633e8fecb0a None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:43.539672 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f334cc3-b1a6-48cb-8e9b-0633e8fecb0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:43.560767 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-7f334cc3-b1a6-48cb-8e9b-0633e8fecb0a None None] MFA Rules not processed for user `34afa4aac10c4a018ca05a0ba34c940a`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:43.562098 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7f334cc3-b1a6-48cb-8e9b-0633e8fecb0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:43.572336 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1119/2240] 10.222.0.64 () {58 vars in 967 bytes} [Tue Oct 4 14:33:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:43.590657 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4ce107-3ae6-4641-8d1b-ea3e5a140e53 service swift] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:43.603650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4ce107-3ae6-4641-8d1b-ea3e5a140e53 service swift] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:43.609862 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f52e01-3fcc-4bd8-ae7a-3c29089a3e85 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5980e353164c4873bcdd2af847e0a1f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:43.611062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f52e01-3fcc-4bd8-ae7a-3c29089a3e85 service swift] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:43.611511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f52e01-3fcc-4bd8-ae7a-3c29089a3e85 service swift] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:43.611948 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f52e01-3fcc-4bd8-ae7a-3c29089a3e85 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:43.623238 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f52e01-3fcc-4bd8-ae7a-3c29089a3e85 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:43.625033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f52e01-3fcc-4bd8-ae7a-3c29089a3e85 service swift] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:43.635262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1122/2241] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Oct 4 14:33:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 14:33:44.000385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb9054d-2af2-4240-88d3-4dd49c155337 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:44.002511 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb9054d-2af2-4240-88d3-4dd49c155337 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:44.002978 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb9054d-2af2-4240-88d3-4dd49c155337 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:44.036916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-ecb9054d-2af2-4240-88d3-4dd49c155337 None None] MFA Rules not processed for user `34afa4aac10c4a018ca05a0ba34c940a`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:44.038475 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ecb9054d-2af2-4240-88d3-4dd49c155337 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:44.053657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1120/2242] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Oct 4 14:33:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:44.061499 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f52e01-3fcc-4bd8-ae7a-3c29089a3e85 service swift] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:44.075713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f52e01-3fcc-4bd8-ae7a-3c29089a3e85 service swift] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:44.085865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7bd659-7771-45db-9fb6-3b683c5f124e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5980e353164c4873bcdd2af847e0a1f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:44.086920 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7bd659-7771-45db-9fb6-3b683c5f124e service swift] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:44.087475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7bd659-7771-45db-9fb6-3b683c5f124e service swift] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:44.088004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7bd659-7771-45db-9fb6-3b683c5f124e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:44.100394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7bd659-7771-45db-9fb6-3b683c5f124e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:44.102420 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7bd659-7771-45db-9fb6-3b683c5f124e service swift] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:44.114856 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1123/2243] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Oct 4 14:33:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 14:33:44.684781 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ffa973-74a5-412d-903c-eff89a07279f None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:44.685494 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ffa973-74a5-412d-903c-eff89a07279f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:44.685965 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ffa973-74a5-412d-903c-eff89a07279f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:44.721222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-a8ffa973-74a5-412d-903c-eff89a07279f None None] MFA Rules not processed for user `34afa4aac10c4a018ca05a0ba34c940a`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:44.722462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a8ffa973-74a5-412d-903c-eff89a07279f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:44.738409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1121/2244] 10.222.0.64 () {58 vars in 967 bytes} [Tue Oct 4 14:33:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:44.752289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7bd659-7771-45db-9fb6-3b683c5f124e service swift] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:44.766573 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7bd659-7771-45db-9fb6-3b683c5f124e service swift] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:44.772602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33a9b71-491d-4564-bc28-cd19c9567ded service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5980e353164c4873bcdd2af847e0a1f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:44.773936 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33a9b71-491d-4564-bc28-cd19c9567ded service swift] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:44.774384 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33a9b71-491d-4564-bc28-cd19c9567ded service swift] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:44.774851 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33a9b71-491d-4564-bc28-cd19c9567ded service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:44.794092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33a9b71-491d-4564-bc28-cd19c9567ded service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:44.795928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33a9b71-491d-4564-bc28-cd19c9567ded service swift] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:44.806248 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1124/2245] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Oct 4 14:33:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 14:33:44.862721 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95d8d7c-527d-4612-bb53-dacf3440fd6e None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:44.862954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95d8d7c-527d-4612-bb53-dacf3440fd6e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:44.863237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95d8d7c-527d-4612-bb53-dacf3440fd6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:44.949886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-e95d8d7c-527d-4612-bb53-dacf3440fd6e None None] MFA Rules not processed for user `cab40292db0d4a74b1fd59c3539e81e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:44.951981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-e95d8d7c-527d-4612-bb53-dacf3440fd6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:45.096091 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1122/2246] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:44 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:45.102106 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33a9b71-491d-4564-bc28-cd19c9567ded service swift] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:45.126665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33a9b71-491d-4564-bc28-cd19c9567ded service swift] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:45.154142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e5f7b8-840c-4268-9816-b0058b08ac5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:45.155617 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e5f7b8-840c-4268-9816-b0058b08ac5a admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:45.156244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e5f7b8-840c-4268-9816-b0058b08ac5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:45.157965 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e5f7b8-840c-4268-9816-b0058b08ac5a admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:45.168398 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e5f7b8-840c-4268-9816-b0058b08ac5a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:45.176985 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e5f7b8-840c-4268-9816-b0058b08ac5a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:45.237479 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1125/2247] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:33:45 2022] POST /identity/v3/projects => generated 381 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:33:45.244209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95d8d7c-527d-4612-bb53-dacf3440fd6e None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:45.273549 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95d8d7c-527d-4612-bb53-dacf3440fd6e None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:45.389369 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aef9950-c6f9-4aa3-9545-68d6ce36334f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:45.390242 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aef9950-c6f9-4aa3-9545-68d6ce36334f admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:45.390619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aef9950-c6f9-4aa3-9545-68d6ce36334f admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:45.390990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aef9950-c6f9-4aa3-9545-68d6ce36334f admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:45.392914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aef9950-c6f9-4aa3-9545-68d6ce36334f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:45.395420 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aef9950-c6f9-4aa3-9545-68d6ce36334f admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:45.424599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1123/2248] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:33:45 2022] POST /identity/v3/users => generated 297 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:33:45.430361 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e5f7b8-840c-4268-9816-b0058b08ac5a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:45.448389 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e5f7b8-840c-4268-9816-b0058b08ac5a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:45.457857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab64129-5be8-4f0f-8da3-a7f1f2055223 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:45.458502 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab64129-5be8-4f0f-8da3-a7f1f2055223 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:45.458784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab64129-5be8-4f0f-8da3-a7f1f2055223 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:45.459060 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab64129-5be8-4f0f-8da3-a7f1f2055223 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:45.459735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab64129-5be8-4f0f-8da3-a7f1f2055223 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:45.461583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab64129-5be8-4f0f-8da3-a7f1f2055223 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:45.477716 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1126/2249] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:33:45 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:33:45.484277 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aef9950-c6f9-4aa3-9545-68d6ce36334f admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:45.502307 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aef9950-c6f9-4aa3-9545-68d6ce36334f admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:45.510703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95644df6-669a-4d6b-9f4f-33ff80ec4bb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:45.511816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95644df6-669a-4d6b-9f4f-33ff80ec4bb8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:45.512287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95644df6-669a-4d6b-9f4f-33ff80ec4bb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:45.512772 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95644df6-669a-4d6b-9f4f-33ff80ec4bb8 admin admin] PATH_INFO: `/v3/projects/941e341c4f564c77a33a6944b6d7bf63/users/724b08c95d8d4386b0ca7c40c5fdb696/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:45.543848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95644df6-669a-4d6b-9f4f-33ff80ec4bb8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=941e341c4f564c77a33a6944b6d7bf63, user_id=724b08c95d8d4386b0ca7c40c5fdb696, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:45.550125 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95644df6-669a-4d6b-9f4f-33ff80ec4bb8 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:45.579196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1124/2250] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:33:45 2022] PUT /identity/v3/projects/941e341c4f564c77a33a6944b6d7bf63/users/724b08c95d8d4386b0ca7c40c5fdb696/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:33:45.584435 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab64129-5be8-4f0f-8da3-a7f1f2055223 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:45.600277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab64129-5be8-4f0f-8da3-a7f1f2055223 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:45.635705 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:45.689627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4bfafa-4b82-477f-81f8-c629914ca2cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:45.690391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4bfafa-4b82-477f-81f8-c629914ca2cb admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:45.690673 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4bfafa-4b82-477f-81f8-c629914ca2cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:45.690949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4bfafa-4b82-477f-81f8-c629914ca2cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:45.691621 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4bfafa-4b82-477f-81f8-c629914ca2cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:45.694031 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4bfafa-4b82-477f-81f8-c629914ca2cb admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:45.718665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1127/2251] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:33:45 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:33:45.718778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95644df6-669a-4d6b-9f4f-33ff80ec4bb8 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:45.729276 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:45.734736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95644df6-669a-4d6b-9f4f-33ff80ec4bb8 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:45.744782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1cf19f-f0e2-48f9-bdb8-4011471b37df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:45.745411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1cf19f-f0e2-48f9-bdb8-4011471b37df admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:45.745690 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1cf19f-f0e2-48f9-bdb8-4011471b37df admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:45.746764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1cf19f-f0e2-48f9-bdb8-4011471b37df admin admin] PATH_INFO: `/v3/projects/941e341c4f564c77a33a6944b6d7bf63/users/724b08c95d8d4386b0ca7c40c5fdb696/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:45.752683 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1cf19f-f0e2-48f9-bdb8-4011471b37df admin admin] RBAC: Authorizing `identity:create_grant(project_id=941e341c4f564c77a33a6944b6d7bf63, user_id=724b08c95d8d4386b0ca7c40c5fdb696, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:45.755716 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1cf19f-f0e2-48f9-bdb8-4011471b37df admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:45.770929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1125/2252] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:33:45 2022] PUT /identity/v3/projects/941e341c4f564c77a33a6944b6d7bf63/users/724b08c95d8d4386b0ca7c40c5fdb696/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:33:47.785289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4bfafa-4b82-477f-81f8-c629914ca2cb admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:47.790795 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0b4bfafa-4b82-477f-81f8-c629914ca2cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:47.814508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4bfafa-4b82-477f-81f8-c629914ca2cb admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:47.911855 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78fb60e1-b403-4682-be85-da5355ffc7bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:47.912624 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fb60e1-b403-4682-be85-da5355ffc7bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:47.913205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fb60e1-b403-4682-be85-da5355ffc7bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:47.913768 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fb60e1-b403-4682-be85-da5355ffc7bb admin admin] PATH_INFO: `/v3/users/b03bae15d7bf43bb871c1a2eed6ae09a` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:47.915376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fb60e1-b403-4682-be85-da5355ffc7bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=b03bae15d7bf43bb871c1a2eed6ae09a)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:47.918562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78fb60e1-b403-4682-be85-da5355ffc7bb admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:48.011292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-78fb60e1-b403-4682-be85-da5355ffc7bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b03bae15d7bf43bb871c1a2eed6ae09a'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:48.021561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-78fb60e1-b403-4682-be85-da5355ffc7bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b03bae15d7bf43bb871c1a2eed6ae09a'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:48.026711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-78fb60e1-b403-4682-be85-da5355ffc7bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b03bae15d7bf43bb871c1a2eed6ae09a'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:48.050321 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-78fb60e1-b403-4682-be85-da5355ffc7bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b03bae15d7bf43bb871c1a2eed6ae09a'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:48.094701 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-78fb60e1-b403-4682-be85-da5355ffc7bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b03bae15d7bf43bb871c1a2eed6ae09a'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:48.163880 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1128/2253] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:33:47 2022] DELETE /identity/v3/users/b03bae15d7bf43bb871c1a2eed6ae09a => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:33:48.175407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e78860-3da8-4a3b-a132-957f1974b814 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:48.175989 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e78860-3da8-4a3b-a132-957f1974b814 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:48.176732 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e78860-3da8-4a3b-a132-957f1974b814 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:48.213742 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-02e78860-3da8-4a3b-a132-957f1974b814 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:48.219959 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-02e78860-3da8-4a3b-a132-957f1974b814 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:48.356409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1126/2254] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:48.369520 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78fb60e1-b403-4682-be85-da5355ffc7bb admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:48.371302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-78fb60e1-b403-4682-be85-da5355ffc7bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:48.462947 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:48.487599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78fb60e1-b403-4682-be85-da5355ffc7bb admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:48.496729 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd044aca-7b0a-4b30-bb8b-73952fb23e70 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:48.497799 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd044aca-7b0a-4b30-bb8b-73952fb23e70 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:48.498294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd044aca-7b0a-4b30-bb8b-73952fb23e70 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:48.498792 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd044aca-7b0a-4b30-bb8b-73952fb23e70 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:48.516029 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd044aca-7b0a-4b30-bb8b-73952fb23e70 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:48.519145 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd044aca-7b0a-4b30-bb8b-73952fb23e70 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:48.541428 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1129/2255] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:33:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:33:48.794963 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e78860-3da8-4a3b-a132-957f1974b814 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:48.796878 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-02e78860-3da8-4a3b-a132-957f1974b814 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:48.812570 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e78860-3da8-4a3b-a132-957f1974b814 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:48.838880 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4beeb3-ef50-4edc-8844-1369d1d2dfd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:48.839694 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4beeb3-ef50-4edc-8844-1369d1d2dfd9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:48.840070 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4beeb3-ef50-4edc-8844-1369d1d2dfd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:48.840595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4beeb3-ef50-4edc-8844-1369d1d2dfd9 admin admin] PATH_INFO: `/v3/projects/f5980ace2a72461689bfb87d48571e72` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:48.844130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4beeb3-ef50-4edc-8844-1369d1d2dfd9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f5980ace2a72461689bfb87d48571e72)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:48.852723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4beeb3-ef50-4edc-8844-1369d1d2dfd9 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:48.962835 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ab4beeb3-ef50-4edc-8844-1369d1d2dfd9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f5980ace2a72461689bfb87d48571e72', 'request_id': 'req-ab4beeb3-ef50-4edc-8844-1369d1d2dfd9'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:48.972477 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ab4beeb3-ef50-4edc-8844-1369d1d2dfd9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f5980ace2a72461689bfb87d48571e72', 'request_id': 'req-ab4beeb3-ef50-4edc-8844-1369d1d2dfd9'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:48.979985 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ab4beeb3-ef50-4edc-8844-1369d1d2dfd9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f5980ace2a72461689bfb87d48571e72', 'request_id': 'req-ab4beeb3-ef50-4edc-8844-1369d1d2dfd9'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:49.000055 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ab4beeb3-ef50-4edc-8844-1369d1d2dfd9 admin admin] The token cache is being invalidate because project f5980ace2a72461689bfb87d48571e72 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:33:49.001101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ab4beeb3-ef50-4edc-8844-1369d1d2dfd9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:49.001984 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ab4beeb3-ef50-4edc-8844-1369d1d2dfd9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:33:49.003532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1127/2256] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:33:48 2022] DELETE /identity/v3/projects/f5980ace2a72461689bfb87d48571e72 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:33:49.008444 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ab8d5a-11c0-4eb5-a409-9746ae63aa4a None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:49.008991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ab8d5a-11c0-4eb5-a409-9746ae63aa4a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:49.008991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ab8d5a-11c0-4eb5-a409-9746ae63aa4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:49.044675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-32ab8d5a-11c0-4eb5-a409-9746ae63aa4a None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:49.046182 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-32ab8d5a-11c0-4eb5-a409-9746ae63aa4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:49.121987 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1130/2257] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:49.131149 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce122bc-bc52-4854-9b91-6b00feeee0be None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:49.131277 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce122bc-bc52-4854-9b91-6b00feeee0be None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:49.131528 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce122bc-bc52-4854-9b91-6b00feeee0be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:49.171307 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-1ce122bc-bc52-4854-9b91-6b00feeee0be None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:49.172710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1ce122bc-bc52-4854-9b91-6b00feeee0be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:49.190789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1128/2258] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:49.197690 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ab8d5a-11c0-4eb5-a409-9746ae63aa4a None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:49.210985 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ab8d5a-11c0-4eb5-a409-9746ae63aa4a None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:49.217675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8503b81d-40c6-4805-9db0-7b7bb1097b6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:49.218709 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8503b81d-40c6-4805-9db0-7b7bb1097b6c admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:49.219216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8503b81d-40c6-4805-9db0-7b7bb1097b6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:49.219709 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8503b81d-40c6-4805-9db0-7b7bb1097b6c admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:49.220839 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8503b81d-40c6-4805-9db0-7b7bb1097b6c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:33:49.221358 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8503b81d-40c6-4805-9db0-7b7bb1097b6c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:49.223193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8503b81d-40c6-4805-9db0-7b7bb1097b6c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:49.240021 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1131/2259] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:33:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:33:49.246909 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce122bc-bc52-4854-9b91-6b00feeee0be None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:49.264997 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce122bc-bc52-4854-9b91-6b00feeee0be None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:49.275589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f47d08-bcbb-48a5-b971-942677d5e9a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:49.277257 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f47d08-bcbb-48a5-b971-942677d5e9a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:49.278038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f47d08-bcbb-48a5-b971-942677d5e9a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:49.278787 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f47d08-bcbb-48a5-b971-942677d5e9a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:49.280478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f47d08-bcbb-48a5-b971-942677d5e9a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:49.290640 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f47d08-bcbb-48a5-b971-942677d5e9a2 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:49.342630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1129/2260] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:33:49 2022] POST /identity/v3/projects => generated 381 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:33:49.347350 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8503b81d-40c6-4805-9db0-7b7bb1097b6c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:49.363794 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8503b81d-40c6-4805-9db0-7b7bb1097b6c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:49.489817 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b520df0-3dab-4f9c-9b18-549fbc6df0a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:49.490083 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b520df0-3dab-4f9c-9b18-549fbc6df0a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:49.490357 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b520df0-3dab-4f9c-9b18-549fbc6df0a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:49.490656 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b520df0-3dab-4f9c-9b18-549fbc6df0a2 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:49.491328 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b520df0-3dab-4f9c-9b18-549fbc6df0a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:49.493503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b520df0-3dab-4f9c-9b18-549fbc6df0a2 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:49.554663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f47d08-bcbb-48a5-b971-942677d5e9a2 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:49.564459 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1132/2261] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:33:49 2022] POST /identity/v3/users => generated 291 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:33:49.574541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f47d08-bcbb-48a5-b971-942677d5e9a2 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:49.582551 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bd2e83-d210-4a58-9dbf-6268fccba249 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:49.583701 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bd2e83-d210-4a58-9dbf-6268fccba249 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:49.585027 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bd2e83-d210-4a58-9dbf-6268fccba249 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:49.585717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bd2e83-d210-4a58-9dbf-6268fccba249 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:49.586903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bd2e83-d210-4a58-9dbf-6268fccba249 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:49.589928 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bd2e83-d210-4a58-9dbf-6268fccba249 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:49.618139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1130/2262] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:33:49 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:33:49.621686 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b520df0-3dab-4f9c-9b18-549fbc6df0a2 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:49.642404 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b520df0-3dab-4f9c-9b18-549fbc6df0a2 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:49.653998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd17cf2-32cd-4de1-9124-48f276e3d739 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:49.654738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd17cf2-32cd-4de1-9124-48f276e3d739 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:49.655224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd17cf2-32cd-4de1-9124-48f276e3d739 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:49.655680 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd17cf2-32cd-4de1-9124-48f276e3d739 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:49.657906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd17cf2-32cd-4de1-9124-48f276e3d739 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:49.660350 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd17cf2-32cd-4de1-9124-48f276e3d739 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:49.678619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1133/2263] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:33:49 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:33:49.684110 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bd2e83-d210-4a58-9dbf-6268fccba249 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:49.739596 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bd2e83-d210-4a58-9dbf-6268fccba249 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:49.747104 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe76ac9-b96b-4947-94cc-89192d9ab696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:49.748394 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe76ac9-b96b-4947-94cc-89192d9ab696 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:49.749060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe76ac9-b96b-4947-94cc-89192d9ab696 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:49.749523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe76ac9-b96b-4947-94cc-89192d9ab696 admin admin] PATH_INFO: `/v3/projects/742e9ad0995f4f96bcc1f75f5689a590/users/6ec52681fc41477e92786ad61aa96a76/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:49.783665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe76ac9-b96b-4947-94cc-89192d9ab696 admin admin] RBAC: Authorizing `identity:create_grant(project_id=742e9ad0995f4f96bcc1f75f5689a590, user_id=6ec52681fc41477e92786ad61aa96a76, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:49.789546 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe76ac9-b96b-4947-94cc-89192d9ab696 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:49.817860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1131/2264] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:33:49 2022] PUT /identity/v3/projects/742e9ad0995f4f96bcc1f75f5689a590/users/6ec52681fc41477e92786ad61aa96a76/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:33:49.825257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec34f1b-2495-40b0-8fdf-7e5e66267c8b None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:49.825675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec34f1b-2495-40b0-8fdf-7e5e66267c8b None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:49.826074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec34f1b-2495-40b0-8fdf-7e5e66267c8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:49.857517 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-0ec34f1b-2495-40b0-8fdf-7e5e66267c8b None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:49.859529 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0ec34f1b-2495-40b0-8fdf-7e5e66267c8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:49.949660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:49.980950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1134/2265] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:49.993635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe76ac9-b96b-4947-94cc-89192d9ab696 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:49.995565 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-dbe76ac9-b96b-4947-94cc-89192d9ab696 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:50.052447 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:50.133083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe76ac9-b96b-4947-94cc-89192d9ab696 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:50.147101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17510ba2-4d4b-40de-a974-64f3f8edeceb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:50.148200 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17510ba2-4d4b-40de-a974-64f3f8edeceb service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:50.148688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17510ba2-4d4b-40de-a974-64f3f8edeceb service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:50.149192 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17510ba2-4d4b-40de-a974-64f3f8edeceb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:50.167091 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17510ba2-4d4b-40de-a974-64f3f8edeceb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:50.169204 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17510ba2-4d4b-40de-a974-64f3f8edeceb service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:50.187898 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1132/2266] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:33:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:33:51.470740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0a7bff-4a22-47a9-9f02-c2bfdd199238 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:51.472186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0a7bff-4a22-47a9-9f02-c2bfdd199238 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:51.472731 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0a7bff-4a22-47a9-9f02-c2bfdd199238 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:51.514552 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-0a0a7bff-4a22-47a9-9f02-c2bfdd199238 None None] MFA Rules not processed for user `6ec52681fc41477e92786ad61aa96a76`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:51.517453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0a0a7bff-4a22-47a9-9f02-c2bfdd199238 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:51.670960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1135/2267] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:51 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:51.680449 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17510ba2-4d4b-40de-a974-64f3f8edeceb service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:51.682455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-17510ba2-4d4b-40de-a974-64f3f8edeceb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:51.803450 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17510ba2-4d4b-40de-a974-64f3f8edeceb service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:51.807564 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aebe973-2f6f-4e51-b65b-555889467a05 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:51.808688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aebe973-2f6f-4e51-b65b-555889467a05 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:51.809908 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aebe973-2f6f-4e51-b65b-555889467a05 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:51.810401 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aebe973-2f6f-4e51-b65b-555889467a05 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:51.826924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aebe973-2f6f-4e51-b65b-555889467a05 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:51.828698 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aebe973-2f6f-4e51-b65b-555889467a05 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:51.842300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1133/2268] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:33:51 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:33:52.843481 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d15093-29da-4c51-b357-78baa0c62a00 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:52.844102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d15093-29da-4c51-b357-78baa0c62a00 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:52.844102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d15093-29da-4c51-b357-78baa0c62a00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:52.884851 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-e6d15093-29da-4c51-b357-78baa0c62a00 None None] MFA Rules not processed for user `724b08c95d8d4386b0ca7c40c5fdb696`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:52.887536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-e6d15093-29da-4c51-b357-78baa0c62a00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:53.018361 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1136/2269] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:52 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:53.034244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aebe973-2f6f-4e51-b65b-555889467a05 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:53.036567 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5aebe973-2f6f-4e51-b65b-555889467a05 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:53.127209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aebe973-2f6f-4e51-b65b-555889467a05 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:53.132302 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72fcccc-3564-47b4-8543-7ac5ac9cf809 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35a5a4ef40204e158649ca25583bbde7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:53.133030 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72fcccc-3564-47b4-8543-7ac5ac9cf809 service glance] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:53.133550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72fcccc-3564-47b4-8543-7ac5ac9cf809 service glance] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:53.134049 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72fcccc-3564-47b4-8543-7ac5ac9cf809 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:53.136281 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a72fcccc-3564-47b4-8543-7ac5ac9cf809 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:53.227663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72fcccc-3564-47b4-8543-7ac5ac9cf809 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:53.229329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72fcccc-3564-47b4-8543-7ac5ac9cf809 service glance] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:53.241046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1134/2270] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:33:53 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:33:53.474746 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee2a467-0725-4107-890b-29272c927265 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:53.475045 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee2a467-0725-4107-890b-29272c927265 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:53.476058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee2a467-0725-4107-890b-29272c927265 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:53.510438 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fcc1cd-507d-4305-be9b-caab6bebff67 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:53.510757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fcc1cd-507d-4305-be9b-caab6bebff67 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:53.511015 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fcc1cd-507d-4305-be9b-caab6bebff67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:53.540284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-86fcc1cd-507d-4305-be9b-caab6bebff67 None None] MFA Rules not processed for user `6ec52681fc41477e92786ad61aa96a76`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:53.541719 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-86fcc1cd-507d-4305-be9b-caab6bebff67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:53.547982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-7ee2a467-0725-4107-890b-29272c927265 None None] MFA Rules not processed for user `34afa4aac10c4a018ca05a0ba34c940a`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:33:53.551156 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7ee2a467-0725-4107-890b-29272c927265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:53.557223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1135/2271] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:33:53 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:33:53.701457 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1137/2272] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Oct 4 14:33:53 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:33:53.721144 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fcc1cd-507d-4305-be9b-caab6bebff67 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:33:53.724587 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-86fcc1cd-507d-4305-be9b-caab6bebff67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:33:53.735761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:33:53.853126 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fcc1cd-507d-4305-be9b-caab6bebff67 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:33:53.859324 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526ea2ee-c5b5-44e0-a2eb-93434d7bd6cb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5980e353164c4873bcdd2af847e0a1f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:33:53.862794 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526ea2ee-c5b5-44e0-a2eb-93434d7bd6cb service swift] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:33:53.864413 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526ea2ee-c5b5-44e0-a2eb-93434d7bd6cb service swift] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:33:53.865733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526ea2ee-c5b5-44e0-a2eb-93434d7bd6cb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:33:53.887343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526ea2ee-c5b5-44e0-a2eb-93434d7bd6cb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:33:53.889526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526ea2ee-c5b5-44e0-a2eb-93434d7bd6cb service swift] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:33:53.902410 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1136/2273] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Oct 4 14:33:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 04 14:34:03.689400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=95 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:34:03.897946 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=31 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:34:08.349349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d64782-a089-4d67-b5bc-e3805d4b1aa4 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:34:08.349928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d64782-a089-4d67-b5bc-e3805d4b1aa4 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:34:08.349928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d64782-a089-4d67-b5bc-e3805d4b1aa4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:34:08.371456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-39d64782-a089-4d67-b5bc-e3805d4b1aa4 None None] MFA Rules not processed for user `cab40292db0d4a74b1fd59c3539e81e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:34:08.372713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-39d64782-a089-4d67-b5bc-e3805d4b1aa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:34:08.381040 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1138/2274] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:34:08 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:34:12.294879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4502541e-cbe2-430b-8679-71d225cb8fd7 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:34:12.295446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4502541e-cbe2-430b-8679-71d225cb8fd7 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:34:12.296537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4502541e-cbe2-430b-8679-71d225cb8fd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:34:12.329229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-4502541e-cbe2-430b-8679-71d225cb8fd7 None None] MFA Rules not processed for user `34afa4aac10c4a018ca05a0ba34c940a`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:34:12.330373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-4502541e-cbe2-430b-8679-71d225cb8fd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:34:12.337834 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1137/2275] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Oct 4 14:34:12 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:34:12.351789 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d64782-a089-4d67-b5bc-e3805d4b1aa4 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:34:12.366292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d64782-a089-4d67-b5bc-e3805d4b1aa4 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:34:12.371749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bc355d-6efd-4b7a-9772-7c2bf9015ef5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5980e353164c4873bcdd2af847e0a1f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:34:12.372935 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bc355d-6efd-4b7a-9772-7c2bf9015ef5 service swift] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:34:12.373217 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bc355d-6efd-4b7a-9772-7c2bf9015ef5 service swift] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:34:12.373657 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bc355d-6efd-4b7a-9772-7c2bf9015ef5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:34:12.391231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31bc355d-6efd-4b7a-9772-7c2bf9015ef5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:34:12.393006 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31bc355d-6efd-4b7a-9772-7c2bf9015ef5 service swift] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:34:12.403696 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1139/2276] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Oct 4 14:34:12 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 14:34:15.570684 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826acc09-a59b-47f2-af9f-180e3779c12b None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:34:15.571657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826acc09-a59b-47f2-af9f-180e3779c12b None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:34:15.572156 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826acc09-a59b-47f2-af9f-180e3779c12b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:34:15.607352 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-826acc09-a59b-47f2-af9f-180e3779c12b None None] MFA Rules not processed for user `54b3856eb61a487dab21d8ffa345701c`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:34:15.610107 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-826acc09-a59b-47f2-af9f-180e3779c12b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:34:15.705738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1138/2277] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:34:15 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:34:22.398393 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=9 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:34:25.682169 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=28 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:34:29.466073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c94770-1201-4752-97bc-c1280e7fcc28 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:34:29.466603 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c94770-1201-4752-97bc-c1280e7fcc28 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:34:29.466694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c94770-1201-4752-97bc-c1280e7fcc28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:34:29.489750 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-50c94770-1201-4752-97bc-c1280e7fcc28 None None] MFA Rules not processed for user `2b4c980bac3e444faa0e6f1a1d270cfa`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:34:29.491166 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-50c94770-1201-4752-97bc-c1280e7fcc28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:34:29.574585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1140/2278] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:34:29 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:34:33.134828 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826acc09-a59b-47f2-af9f-180e3779c12b None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:34:33.136943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-826acc09-a59b-47f2-af9f-180e3779c12b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:34:33.151330 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826acc09-a59b-47f2-af9f-180e3779c12b None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:34:33.168484 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09064db9-5dbf-4329-8385-6d57a258fed0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:34:33.169196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09064db9-5dbf-4329-8385-6d57a258fed0 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:34:33.169537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09064db9-5dbf-4329-8385-6d57a258fed0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:34:33.172023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09064db9-5dbf-4329-8385-6d57a258fed0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:34:33.172800 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-09064db9-5dbf-4329-8385-6d57a258fed0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:34:33.262253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09064db9-5dbf-4329-8385-6d57a258fed0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:34:33.263695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09064db9-5dbf-4329-8385-6d57a258fed0 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:34:33.276296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1139/2279] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:34:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:34:34.412700 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c94770-1201-4752-97bc-c1280e7fcc28 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:34:34.423694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c94770-1201-4752-97bc-c1280e7fcc28 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:34:34.427307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d7e1a9-4a5f-4b9d-9d61-8160a0cf0061 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:34:34.429107 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d7e1a9-4a5f-4b9d-9d61-8160a0cf0061 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:34:34.429107 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d7e1a9-4a5f-4b9d-9d61-8160a0cf0061 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:34:34.429107 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d7e1a9-4a5f-4b9d-9d61-8160a0cf0061 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:34:34.430581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a9d7e1a9-4a5f-4b9d-9d61-8160a0cf0061 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:34:34.449780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d7e1a9-4a5f-4b9d-9d61-8160a0cf0061 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:34:34.455244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d7e1a9-4a5f-4b9d-9d61-8160a0cf0061 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:34:34.471004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1141/2280] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:34:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:34:40.486864 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16583106-0274-4d26-9420-f426d1e107a8 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:34:40.487410 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16583106-0274-4d26-9420-f426d1e107a8 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:34:40.487532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16583106-0274-4d26-9420-f426d1e107a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:34:40.508068 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-16583106-0274-4d26-9420-f426d1e107a8 None None] MFA Rules not processed for user `c5b38375d6ff4d0db0f3f7bc89ca8798`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:34:40.509637 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-16583106-0274-4d26-9420-f426d1e107a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:34:40.571242 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1140/2281] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:34:40 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:34:44.465303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=31 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:34:47.215440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d7e1a9-4a5f-4b9d-9d61-8160a0cf0061 service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:34:47.225496 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d7e1a9-4a5f-4b9d-9d61-8160a0cf0061 service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:34:47.229315 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716a58c9-d84e-4730-a6f6-10fb85563e5f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:34:47.229925 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716a58c9-d84e-4730-a6f6-10fb85563e5f service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:34:47.230205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716a58c9-d84e-4730-a6f6-10fb85563e5f service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:34:47.230542 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716a58c9-d84e-4730-a6f6-10fb85563e5f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:34:47.232668 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-716a58c9-d84e-4730-a6f6-10fb85563e5f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:34:47.244272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716a58c9-d84e-4730-a6f6-10fb85563e5f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:34:47.245751 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716a58c9-d84e-4730-a6f6-10fb85563e5f service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:34:47.258251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1142/2282] 10.222.0.64 () {60 vars in 1396 bytes} [Tue Oct 4 14:34:47 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3262 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:34:50.564524 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=59 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:34:52.614448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16583106-0274-4d26-9420-f426d1e107a8 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:34:52.616095 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-16583106-0274-4d26-9420-f426d1e107a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:34:52.633874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16583106-0274-4d26-9420-f426d1e107a8 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:34:52.639956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f43ae12-8d16-4d76-bf29-19162202517a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:34:52.640622 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f43ae12-8d16-4d76-bf29-19162202517a service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:34:52.640902 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f43ae12-8d16-4d76-bf29-19162202517a service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:34:52.641175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f43ae12-8d16-4d76-bf29-19162202517a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:34:52.643424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1f43ae12-8d16-4d76-bf29-19162202517a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:34:52.656243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f43ae12-8d16-4d76-bf29-19162202517a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:34:52.657812 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f43ae12-8d16-4d76-bf29-19162202517a service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:34:52.668595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1141/2283] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:34:52 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:34:57.252243 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=6 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:34:59.262439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5291eb3-b701-4e1d-b092-245fc949caf5 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:34:59.263092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5291eb3-b701-4e1d-b092-245fc949caf5 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:34:59.263856 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5291eb3-b701-4e1d-b092-245fc949caf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:34:59.283960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-a5291eb3-b701-4e1d-b092-245fc949caf5 None None] MFA Rules not processed for user `fc02bb4b8a32400faef8743d2418a648`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:34:59.285204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a5291eb3-b701-4e1d-b092-245fc949caf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:34:59.353721 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1143/2284] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:34:59 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:34:59.708052 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f43ae12-8d16-4d76-bf29-19162202517a service nova] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:34:59.718576 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f43ae12-8d16-4d76-bf29-19162202517a service nova] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:34:59.722211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7399e93f-0476-499c-9c0a-4dd4ac3f5385 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:34:59.722860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7399e93f-0476-499c-9c0a-4dd4ac3f5385 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:34:59.723142 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7399e93f-0476-499c-9c0a-4dd4ac3f5385 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:34:59.723412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7399e93f-0476-499c-9c0a-4dd4ac3f5385 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:34:59.725211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7399e93f-0476-499c-9c0a-4dd4ac3f5385 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:34:59.736230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7399e93f-0476-499c-9c0a-4dd4ac3f5385 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:34:59.737760 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7399e93f-0476-499c-9c0a-4dd4ac3f5385 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:34:59.746125 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1142/2285] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:34:59 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:35:00.112970 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5291eb3-b701-4e1d-b092-245fc949caf5 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:35:00.117718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a5291eb3-b701-4e1d-b092-245fc949caf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:35:00.209848 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5291eb3-b701-4e1d-b092-245fc949caf5 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:35:00.214703 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b949208a-0f49-4f97-9d74-54f62eec472c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '436deb3d60c54a799700986519b2ef68', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:35:00.215313 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b949208a-0f49-4f97-9d74-54f62eec472c service placement] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:35:00.215598 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b949208a-0f49-4f97-9d74-54f62eec472c service placement] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:35:00.215874 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b949208a-0f49-4f97-9d74-54f62eec472c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:35:00.219196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-b949208a-0f49-4f97-9d74-54f62eec472c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:35:00.230900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b949208a-0f49-4f97-9d74-54f62eec472c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:35:00.232579 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b949208a-0f49-4f97-9d74-54f62eec472c service placement] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:35:00.242781 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1144/2286] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:35:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:35:00.776599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7399e93f-0476-499c-9c0a-4dd4ac3f5385 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:35:00.786355 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7399e93f-0476-499c-9c0a-4dd4ac3f5385 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:35:00.791710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c840e6-174b-458e-b441-1f443649e649 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:35:00.792331 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c840e6-174b-458e-b441-1f443649e649 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:35:00.792688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c840e6-174b-458e-b441-1f443649e649 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:35:00.792954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c840e6-174b-458e-b441-1f443649e649 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:35:00.803534 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c840e6-174b-458e-b441-1f443649e649 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:35:00.805210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c840e6-174b-458e-b441-1f443649e649 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:35:00.815936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1143/2287] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:35:00 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:35:01.276969 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b949208a-0f49-4f97-9d74-54f62eec472c service placement] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:35:01.293224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b949208a-0f49-4f97-9d74-54f62eec472c service placement] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:35:01.299984 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d51223-8543-417a-9905-02a9a6bd5080 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:35:01.300799 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d51223-8543-417a-9905-02a9a6bd5080 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:35:01.301468 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d51223-8543-417a-9905-02a9a6bd5080 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:35:01.301985 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d51223-8543-417a-9905-02a9a6bd5080 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:35:01.305202 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a1d51223-8543-417a-9905-02a9a6bd5080 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:35:01.322516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d51223-8543-417a-9905-02a9a6bd5080 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:35:01.324489 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d51223-8543-417a-9905-02a9a6bd5080 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:35:01.336074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1145/2288] 10.222.0.64 () {60 vars in 1396 bytes} [Tue Oct 4 14:35:01 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3252 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:35:07.273701 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c840e6-174b-458e-b441-1f443649e649 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:35:07.284385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c840e6-174b-458e-b441-1f443649e649 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:35:07.288423 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b5342b-6299-4181-aae0-99a2fbd4c859 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:35:07.289121 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b5342b-6299-4181-aae0-99a2fbd4c859 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:35:07.289467 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b5342b-6299-4181-aae0-99a2fbd4c859 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:35:07.289827 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b5342b-6299-4181-aae0-99a2fbd4c859 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:35:07.292167 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-24b5342b-6299-4181-aae0-99a2fbd4c859 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:35:07.305109 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b5342b-6299-4181-aae0-99a2fbd4c859 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:35:07.306815 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b5342b-6299-4181-aae0-99a2fbd4c859 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:35:07.318248 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1144/2289] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:35:07 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:35:11.331475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=63 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:35:17.303594 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72afa043-205f-4cb9-b802-1859ed6e777a None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:35:17.304287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72afa043-205f-4cb9-b802-1859ed6e777a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:35:17.304287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72afa043-205f-4cb9-b802-1859ed6e777a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:35:17.312741 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=24 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:35:17.327218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-72afa043-205f-4cb9-b802-1859ed6e777a None None] MFA Rules not processed for user `6ec52681fc41477e92786ad61aa96a76`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:35:17.328499 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-72afa043-205f-4cb9-b802-1859ed6e777a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:35:17.335586 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1146/2290] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:35:17 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:35:19.343708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b5342b-6299-4181-aae0-99a2fbd4c859 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:35:19.358019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b5342b-6299-4181-aae0-99a2fbd4c859 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:35:19.366597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091ddd13-f3fd-4523-bff2-1cfe6e61cb0e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:35:19.367562 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091ddd13-f3fd-4523-bff2-1cfe6e61cb0e service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:35:19.367932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091ddd13-f3fd-4523-bff2-1cfe6e61cb0e service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:35:19.368601 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091ddd13-f3fd-4523-bff2-1cfe6e61cb0e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:35:19.370315 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-091ddd13-f3fd-4523-bff2-1cfe6e61cb0e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:35:19.388129 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091ddd13-f3fd-4523-bff2-1cfe6e61cb0e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:35:19.390052 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091ddd13-f3fd-4523-bff2-1cfe6e61cb0e service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:35:19.402396 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1145/2291] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:35:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:35:23.255704 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a721c0bb-4d4d-43e7-9d32-d5f0b2a88cb6 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:35:23.256779 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a721c0bb-4d4d-43e7-9d32-d5f0b2a88cb6 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:35:23.257264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a721c0bb-4d4d-43e7-9d32-d5f0b2a88cb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:35:23.279545 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-a721c0bb-4d4d-43e7-9d32-d5f0b2a88cb6 None None] MFA Rules not processed for user `c7028d2ccede4b44b567387bb83c4579`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:35:23.281640 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a721c0bb-4d4d-43e7-9d32-d5f0b2a88cb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:35:23.370687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1147/2292] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:35:23 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:35:23.450936 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091ddd13-f3fd-4523-bff2-1cfe6e61cb0e service nova] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:35:23.452336 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-091ddd13-f3fd-4523-bff2-1cfe6e61cb0e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:35:23.468594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091ddd13-f3fd-4523-bff2-1cfe6e61cb0e service nova] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:35:23.480353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e76abe-ddd4-44be-8c4f-c4d1be876712 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:35:23.482730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e76abe-ddd4-44be-8c4f-c4d1be876712 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:35:23.483509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e76abe-ddd4-44be-8c4f-c4d1be876712 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:35:23.484600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e76abe-ddd4-44be-8c4f-c4d1be876712 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:35:23.490048 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d5e76abe-ddd4-44be-8c4f-c4d1be876712 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:35:23.511601 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e76abe-ddd4-44be-8c4f-c4d1be876712 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:35:23.529543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e76abe-ddd4-44be-8c4f-c4d1be876712 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:35:23.572363 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1146/2293] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:35:23 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:35:23.960108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a721c0bb-4d4d-43e7-9d32-d5f0b2a88cb6 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:35:23.976102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a721c0bb-4d4d-43e7-9d32-d5f0b2a88cb6 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:35:23.984971 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08c74f5-0c9a-42ea-b90e-83184e640212 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:35:23.986040 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08c74f5-0c9a-42ea-b90e-83184e640212 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:35:23.986546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08c74f5-0c9a-42ea-b90e-83184e640212 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:35:23.986971 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08c74f5-0c9a-42ea-b90e-83184e640212 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:35:23.989537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d08c74f5-0c9a-42ea-b90e-83184e640212 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:35:24.011570 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08c74f5-0c9a-42ea-b90e-83184e640212 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:35:24.026014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08c74f5-0c9a-42ea-b90e-83184e640212 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:35:24.039022 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1148/2294] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:35:23 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:35:33.159971 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e76abe-ddd4-44be-8c4f-c4d1be876712 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:35:33.162993 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d5e76abe-ddd4-44be-8c4f-c4d1be876712 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:35:33.202051 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e76abe-ddd4-44be-8c4f-c4d1be876712 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:35:33.220559 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bfb1e6-9960-4a7d-81c4-cb9fd0f5e661 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:35:33.221763 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bfb1e6-9960-4a7d-81c4-cb9fd0f5e661 service nova] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:35:33.224248 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bfb1e6-9960-4a7d-81c4-cb9fd0f5e661 service nova] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:35:33.224692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bfb1e6-9960-4a7d-81c4-cb9fd0f5e661 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:35:33.230782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-01bfb1e6-9960-4a7d-81c4-cb9fd0f5e661 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:35:33.254892 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bfb1e6-9960-4a7d-81c4-cb9fd0f5e661 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:35:33.257667 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bfb1e6-9960-4a7d-81c4-cb9fd0f5e661 service nova] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:35:33.292116 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1147/2295] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:35:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:35:34.034925 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=34 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:35:43.266234 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=18 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:35:52.602452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08c74f5-0c9a-42ea-b90e-83184e640212 service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:35:52.622963 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08c74f5-0c9a-42ea-b90e-83184e640212 service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:35:52.630904 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740ad3ce-3861-4735-8543-47765dc19aa4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:35:52.631597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740ad3ce-3861-4735-8543-47765dc19aa4 service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:35:52.631880 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740ad3ce-3861-4735-8543-47765dc19aa4 service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:35:52.632210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740ad3ce-3861-4735-8543-47765dc19aa4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:35:52.635026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-740ad3ce-3861-4735-8543-47765dc19aa4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:35:52.658153 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740ad3ce-3861-4735-8543-47765dc19aa4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:35:52.660693 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740ad3ce-3861-4735-8543-47765dc19aa4 service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:35:52.682176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1149/2296] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:35:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:35:58.093357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bfb1e6-9960-4a7d-81c4-cb9fd0f5e661 service nova] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:35:58.107874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bfb1e6-9960-4a7d-81c4-cb9fd0f5e661 service nova] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:35:58.113061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71adf9b-5019-4e55-b6eb-ab78615aef26 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:35:58.113672 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71adf9b-5019-4e55-b6eb-ab78615aef26 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:35:58.113998 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71adf9b-5019-4e55-b6eb-ab78615aef26 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:35:58.114219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71adf9b-5019-4e55-b6eb-ab78615aef26 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:35:58.117610 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d71adf9b-5019-4e55-b6eb-ab78615aef26 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:35:58.146523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71adf9b-5019-4e55-b6eb-ab78615aef26 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:35:58.149576 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71adf9b-5019-4e55-b6eb-ab78615aef26 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:35:58.167673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1148/2297] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:35:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 76 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:36:02.676837 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=6 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:03.115727 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740ad3ce-3861-4735-8543-47765dc19aa4 service nova] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:03.119487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-740ad3ce-3861-4735-8543-47765dc19aa4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:03.271148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740ad3ce-3861-4735-8543-47765dc19aa4 service nova] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:03.279739 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86efb88b-24af-414c-a296-8dba94e0ed03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:03.281078 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86efb88b-24af-414c-a296-8dba94e0ed03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:03.281955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86efb88b-24af-414c-a296-8dba94e0ed03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:03.282421 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86efb88b-24af-414c-a296-8dba94e0ed03 admin admin] PATH_INFO: `/v3/users/2b4c980bac3e444faa0e6f1a1d270cfa` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:03.284120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86efb88b-24af-414c-a296-8dba94e0ed03 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2b4c980bac3e444faa0e6f1a1d270cfa)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:03.286545 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86efb88b-24af-414c-a296-8dba94e0ed03 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:03.349796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-86efb88b-24af-414c-a296-8dba94e0ed03 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2b4c980bac3e444faa0e6f1a1d270cfa'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:03.350565 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-86efb88b-24af-414c-a296-8dba94e0ed03 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2b4c980bac3e444faa0e6f1a1d270cfa'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:03.351337 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-86efb88b-24af-414c-a296-8dba94e0ed03 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2b4c980bac3e444faa0e6f1a1d270cfa'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:03.363352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-86efb88b-24af-414c-a296-8dba94e0ed03 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2b4c980bac3e444faa0e6f1a1d270cfa'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:03.387582 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-86efb88b-24af-414c-a296-8dba94e0ed03 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2b4c980bac3e444faa0e6f1a1d270cfa'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:03.408168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1150/2298] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:36:03 2022] DELETE /identity/v3/users/2b4c980bac3e444faa0e6f1a1d270cfa => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:36:03.643905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71adf9b-5019-4e55-b6eb-ab78615aef26 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:03.726699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71adf9b-5019-4e55-b6eb-ab78615aef26 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:03.810690 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072b36f3-f037-4451-b59c-1f171c993d79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:03.811238 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b36f3-f037-4451-b59c-1f171c993d79 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:03.811761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b36f3-f037-4451-b59c-1f171c993d79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:03.811761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b36f3-f037-4451-b59c-1f171c993d79 admin admin] PATH_INFO: `/v3/projects/ec24f1b720bd42ebb37a36d65757aacd` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:03.815219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072b36f3-f037-4451-b59c-1f171c993d79 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ec24f1b720bd42ebb37a36d65757aacd)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:03.815525 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072b36f3-f037-4451-b59c-1f171c993d79 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:03.900387 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-072b36f3-f037-4451-b59c-1f171c993d79 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ec24f1b720bd42ebb37a36d65757aacd', 'request_id': 'req-072b36f3-f037-4451-b59c-1f171c993d79'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:03.910648 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-072b36f3-f037-4451-b59c-1f171c993d79 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ec24f1b720bd42ebb37a36d65757aacd', 'request_id': 'req-072b36f3-f037-4451-b59c-1f171c993d79'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:03.921882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-072b36f3-f037-4451-b59c-1f171c993d79 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ec24f1b720bd42ebb37a36d65757aacd', 'request_id': 'req-072b36f3-f037-4451-b59c-1f171c993d79'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:03.946431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-072b36f3-f037-4451-b59c-1f171c993d79 admin admin] The token cache is being invalidate because project ec24f1b720bd42ebb37a36d65757aacd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:36:03.946752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-072b36f3-f037-4451-b59c-1f171c993d79 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:03.947510 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-072b36f3-f037-4451-b59c-1f171c993d79 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:03.949268 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1149/2299] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:36:03 2022] DELETE /identity/v3/projects/ec24f1b720bd42ebb37a36d65757aacd => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:03.961621 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddbc22a-fc5b-450c-a6da-ab423223490f None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:03.962253 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddbc22a-fc5b-450c-a6da-ab423223490f None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:03.962253 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddbc22a-fc5b-450c-a6da-ab423223490f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:04.071551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-eddbc22a-fc5b-450c-a6da-ab423223490f None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:04.072899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-eddbc22a-fc5b-450c-a6da-ab423223490f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:04.126502 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:04.159555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1151/2300] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:04.173036 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6578ce38-2bbb-4633-8ce2-3d8024210eb5 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:04.173527 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6578ce38-2bbb-4633-8ce2-3d8024210eb5 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:04.173974 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6578ce38-2bbb-4633-8ce2-3d8024210eb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:04.201926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-6578ce38-2bbb-4633-8ce2-3d8024210eb5 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:04.203204 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6578ce38-2bbb-4633-8ce2-3d8024210eb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:04.211487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1150/2301] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:04.216179 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddbc22a-fc5b-450c-a6da-ab423223490f None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:04.228289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddbc22a-fc5b-450c-a6da-ab423223490f None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:04.234311 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decfa1ad-db8c-47d8-be9c-eecf3419719f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:04.234948 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decfa1ad-db8c-47d8-be9c-eecf3419719f admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:04.235237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decfa1ad-db8c-47d8-be9c-eecf3419719f admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:04.235517 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decfa1ad-db8c-47d8-be9c-eecf3419719f admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:04.236182 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decfa1ad-db8c-47d8-be9c-eecf3419719f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:36:04.236519 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decfa1ad-db8c-47d8-be9c-eecf3419719f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:04.238146 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decfa1ad-db8c-47d8-be9c-eecf3419719f admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:04.252158 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1152/2302] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:36:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:36:04.258313 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6578ce38-2bbb-4633-8ce2-3d8024210eb5 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:04.268789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6578ce38-2bbb-4633-8ce2-3d8024210eb5 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:04.275676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfb35e9-8315-4257-ab62-a0ef5e2323c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:04.276336 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfb35e9-8315-4257-ab62-a0ef5e2323c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:04.276616 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfb35e9-8315-4257-ab62-a0ef5e2323c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:04.276883 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfb35e9-8315-4257-ab62-a0ef5e2323c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:04.277792 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfb35e9-8315-4257-ab62-a0ef5e2323c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:04.279967 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfb35e9-8315-4257-ab62-a0ef5e2323c6 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:04.315672 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:04.327730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1151/2303] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:36:04 2022] POST /identity/v3/projects => generated 379 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:36:04.328873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decfa1ad-db8c-47d8-be9c-eecf3419719f admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:04.339493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decfa1ad-db8c-47d8-be9c-eecf3419719f admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:04.424900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17d1e2a-1c13-490e-9321-35b34f715680 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:04.425655 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17d1e2a-1c13-490e-9321-35b34f715680 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:04.426078 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17d1e2a-1c13-490e-9321-35b34f715680 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:04.426461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17d1e2a-1c13-490e-9321-35b34f715680 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:04.427216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17d1e2a-1c13-490e-9321-35b34f715680 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:04.429671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17d1e2a-1c13-490e-9321-35b34f715680 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:04.457636 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1153/2304] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:36:04 2022] POST /identity/v3/users => generated 290 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:36:04.463141 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfb35e9-8315-4257-ab62-a0ef5e2323c6 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:04.475606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfb35e9-8315-4257-ab62-a0ef5e2323c6 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:04.481919 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e332dd77-557b-4457-98b4-51774bd36cbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:04.482548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e332dd77-557b-4457-98b4-51774bd36cbd admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:04.482831 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e332dd77-557b-4457-98b4-51774bd36cbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:04.483094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e332dd77-557b-4457-98b4-51774bd36cbd admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:04.483759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e332dd77-557b-4457-98b4-51774bd36cbd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:04.485439 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e332dd77-557b-4457-98b4-51774bd36cbd admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:04.502786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1152/2305] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:36:04 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:36:04.504082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17d1e2a-1c13-490e-9321-35b34f715680 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:04.516693 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17d1e2a-1c13-490e-9321-35b34f715680 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:04.523266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794db95e-2cbe-4d99-8c59-e8e04fd6e2f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:04.523863 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794db95e-2cbe-4d99-8c59-e8e04fd6e2f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:04.524228 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794db95e-2cbe-4d99-8c59-e8e04fd6e2f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:04.524472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794db95e-2cbe-4d99-8c59-e8e04fd6e2f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:04.525174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794db95e-2cbe-4d99-8c59-e8e04fd6e2f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:04.526825 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794db95e-2cbe-4d99-8c59-e8e04fd6e2f9 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:04.541835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1154/2306] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:36:04 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:36:04.549126 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e332dd77-557b-4457-98b4-51774bd36cbd admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:04.562154 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e332dd77-557b-4457-98b4-51774bd36cbd admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:04.567742 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1acf71-8120-4823-9f3b-735559d8a987 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:04.568559 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1acf71-8120-4823-9f3b-735559d8a987 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:04.568943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1acf71-8120-4823-9f3b-735559d8a987 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:04.569262 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1acf71-8120-4823-9f3b-735559d8a987 admin admin] PATH_INFO: `/v3/projects/1e77d61e02534817a1a37b83a01a0fb4/users/7274fc324ce044c2b35541621fc5ec7d/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:04.592042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1acf71-8120-4823-9f3b-735559d8a987 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e77d61e02534817a1a37b83a01a0fb4, user_id=7274fc324ce044c2b35541621fc5ec7d, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:04.597142 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1acf71-8120-4823-9f3b-735559d8a987 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:04.623409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1153/2307] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:36:04 2022] PUT /identity/v3/projects/1e77d61e02534817a1a37b83a01a0fb4/users/7274fc324ce044c2b35541621fc5ec7d/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:36:04.642098 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0060c748-eb5a-4f44-a746-f099698bf30b None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:04.642379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0060c748-eb5a-4f44-a746-f099698bf30b None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:04.642654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0060c748-eb5a-4f44-a746-f099698bf30b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:04.681462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-0060c748-eb5a-4f44-a746-f099698bf30b None None] MFA Rules not processed for user `7274fc324ce044c2b35541621fc5ec7d`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:04.682707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0060c748-eb5a-4f44-a746-f099698bf30b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:04.792487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1155/2308] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:04 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:04.800331 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1acf71-8120-4823-9f3b-735559d8a987 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:04.876696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1acf71-8120-4823-9f3b-735559d8a987 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:04.883419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e896677-fcd3-4b70-83d7-1c98d69a90d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:04.884519 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e896677-fcd3-4b70-83d7-1c98d69a90d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:04.885009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e896677-fcd3-4b70-83d7-1c98d69a90d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:04.885481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e896677-fcd3-4b70-83d7-1c98d69a90d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:04.887223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e896677-fcd3-4b70-83d7-1c98d69a90d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:04.890414 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e896677-fcd3-4b70-83d7-1c98d69a90d9 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:04.924642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1154/2309] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:36:04 2022] POST /identity/v3/projects => generated 381 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:36:04.930810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0060c748-eb5a-4f44-a746-f099698bf30b None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:04.946086 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0060c748-eb5a-4f44-a746-f099698bf30b None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:04.999719 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:05.032208 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4023639-56d9-4b0c-b35a-4ba2d65af6d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:05.032873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4023639-56d9-4b0c-b35a-4ba2d65af6d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:05.033148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4023639-56d9-4b0c-b35a-4ba2d65af6d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:05.033423 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4023639-56d9-4b0c-b35a-4ba2d65af6d1 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:05.034089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4023639-56d9-4b0c-b35a-4ba2d65af6d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:05.036100 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4023639-56d9-4b0c-b35a-4ba2d65af6d1 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:05.072359 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1156/2310] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:36:04 2022] POST /identity/v3/users => generated 297 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:36:05.077167 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e896677-fcd3-4b70-83d7-1c98d69a90d9 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:05.097303 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e896677-fcd3-4b70-83d7-1c98d69a90d9 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:05.104537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d713ff9b-4db9-42a6-acb7-2ca483e91e4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:05.105992 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d713ff9b-4db9-42a6-acb7-2ca483e91e4e admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:05.106545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d713ff9b-4db9-42a6-acb7-2ca483e91e4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:05.107079 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d713ff9b-4db9-42a6-acb7-2ca483e91e4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:05.108293 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d713ff9b-4db9-42a6-acb7-2ca483e91e4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:05.111280 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d713ff9b-4db9-42a6-acb7-2ca483e91e4e admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:05.138382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1155/2311] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:36:05 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:36:05.139503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4023639-56d9-4b0c-b35a-4ba2d65af6d1 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:05.150380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4023639-56d9-4b0c-b35a-4ba2d65af6d1 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:05.156978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6228e8-7531-4155-9b73-1ba0ae44c4d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:05.158040 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6228e8-7531-4155-9b73-1ba0ae44c4d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:05.158317 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6228e8-7531-4155-9b73-1ba0ae44c4d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:05.158585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6228e8-7531-4155-9b73-1ba0ae44c4d0 admin admin] PATH_INFO: `/v3/projects/b7f36be12d394ea6abafee8630aed906/users/7a24d295879847219a68d38ee5b59633/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:05.178084 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6228e8-7531-4155-9b73-1ba0ae44c4d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b7f36be12d394ea6abafee8630aed906, user_id=7a24d295879847219a68d38ee5b59633, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:05.180954 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6228e8-7531-4155-9b73-1ba0ae44c4d0 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:05.197072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1157/2312] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:36:05 2022] PUT /identity/v3/projects/b7f36be12d394ea6abafee8630aed906/users/7a24d295879847219a68d38ee5b59633/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:05.204657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d713ff9b-4db9-42a6-acb7-2ca483e91e4e admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:05.218371 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d713ff9b-4db9-42a6-acb7-2ca483e91e4e admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:05.330015 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1caa823b-9ab2-4f89-8995-ac98fe0d49a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:05.330995 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1caa823b-9ab2-4f89-8995-ac98fe0d49a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:05.331482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1caa823b-9ab2-4f89-8995-ac98fe0d49a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:05.331964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1caa823b-9ab2-4f89-8995-ac98fe0d49a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:05.333184 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1caa823b-9ab2-4f89-8995-ac98fe0d49a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:05.336243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1caa823b-9ab2-4f89-8995-ac98fe0d49a5 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:05.357354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1156/2313] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:36:05 2022] GET /identity/v3/roles => generated 1420 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:36:05.361633 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6228e8-7531-4155-9b73-1ba0ae44c4d0 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:05.372650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6228e8-7531-4155-9b73-1ba0ae44c4d0 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:05.377648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5c8326-0b9f-4f87-9b88-209022164547 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:05.378562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5c8326-0b9f-4f87-9b88-209022164547 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:05.378846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5c8326-0b9f-4f87-9b88-209022164547 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:05.379345 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5c8326-0b9f-4f87-9b88-209022164547 admin admin] PATH_INFO: `/v3/projects/b7f36be12d394ea6abafee8630aed906/users/7a24d295879847219a68d38ee5b59633/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:05.381510 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5c8326-0b9f-4f87-9b88-209022164547 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b7f36be12d394ea6abafee8630aed906, user_id=7a24d295879847219a68d38ee5b59633, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:05.384334 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5c8326-0b9f-4f87-9b88-209022164547 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:05.403402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1158/2314] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:36:05 2022] PUT /identity/v3/projects/b7f36be12d394ea6abafee8630aed906/users/7a24d295879847219a68d38ee5b59633/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:36:05.414653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7132e176-16dd-49de-a68d-7158f90445a7 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:05.414940 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7132e176-16dd-49de-a68d-7158f90445a7 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:05.415219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7132e176-16dd-49de-a68d-7158f90445a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:05.446214 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-7132e176-16dd-49de-a68d-7158f90445a7 None None] MFA Rules not processed for user `7a24d295879847219a68d38ee5b59633`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:05.447472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7132e176-16dd-49de-a68d-7158f90445a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:05.484288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:05.557556 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1157/2315] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:05 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:05.572666 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5c8326-0b9f-4f87-9b88-209022164547 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:05.575020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-2b5c8326-0b9f-4f87-9b88-209022164547 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:05.684826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5c8326-0b9f-4f87-9b88-209022164547 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:05.688693 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31c53ca-3bb0-41da-a976-55b55327694c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:05.689367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31c53ca-3bb0-41da-a976-55b55327694c service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:05.689718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31c53ca-3bb0-41da-a976-55b55327694c service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:05.690039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31c53ca-3bb0-41da-a976-55b55327694c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:05.700449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31c53ca-3bb0-41da-a976-55b55327694c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:05.701929 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31c53ca-3bb0-41da-a976-55b55327694c service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:05.712630 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1159/2316] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:36:05 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:36:05.798904 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7132e176-16dd-49de-a68d-7158f90445a7 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:05.811282 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7132e176-16dd-49de-a68d-7158f90445a7 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:05.816693 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674ff311-6b2d-4c69-be19-608183750e09 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:05.817418 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674ff311-6b2d-4c69-be19-608183750e09 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:05.817698 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674ff311-6b2d-4c69-be19-608183750e09 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:05.818084 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674ff311-6b2d-4c69-be19-608183750e09 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:05.914764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674ff311-6b2d-4c69-be19-608183750e09 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:05.917587 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674ff311-6b2d-4c69-be19-608183750e09 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:05.932928 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1158/2317] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:36:05 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:36:12.017280 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdde10df-4072-4b5a-9036-ee9bfa809f28 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:12.017875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdde10df-4072-4b5a-9036-ee9bfa809f28 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:12.018253 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdde10df-4072-4b5a-9036-ee9bfa809f28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:12.052391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-cdde10df-4072-4b5a-9036-ee9bfa809f28 None None] MFA Rules not processed for user `54b3856eb61a487dab21d8ffa345701c`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:12.054570 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-cdde10df-4072-4b5a-9036-ee9bfa809f28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:12.119054 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:12.127990 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1160/2318] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:12 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:15.925829 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=46 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:18.521018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674ff311-6b2d-4c69-be19-608183750e09 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:18.599534 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674ff311-6b2d-4c69-be19-608183750e09 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:18.604402 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c583e5-f89d-4a27-a1d7-410fab1fa2b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:18.605741 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c583e5-f89d-4a27-a1d7-410fab1fa2b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:18.606005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c583e5-f89d-4a27-a1d7-410fab1fa2b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:18.606517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c583e5-f89d-4a27-a1d7-410fab1fa2b8 admin admin] PATH_INFO: `/v3/users/7274fc324ce044c2b35541621fc5ec7d` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:18.607681 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c583e5-f89d-4a27-a1d7-410fab1fa2b8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7274fc324ce044c2b35541621fc5ec7d)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:18.609848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c583e5-f89d-4a27-a1d7-410fab1fa2b8 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:18.659220 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c1c583e5-f89d-4a27-a1d7-410fab1fa2b8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7274fc324ce044c2b35541621fc5ec7d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:18.666906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c1c583e5-f89d-4a27-a1d7-410fab1fa2b8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7274fc324ce044c2b35541621fc5ec7d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:18.674854 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c1c583e5-f89d-4a27-a1d7-410fab1fa2b8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7274fc324ce044c2b35541621fc5ec7d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:18.686803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c1c583e5-f89d-4a27-a1d7-410fab1fa2b8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7274fc324ce044c2b35541621fc5ec7d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:18.687498 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c1c583e5-f89d-4a27-a1d7-410fab1fa2b8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7274fc324ce044c2b35541621fc5ec7d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:18.700990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1159/2319] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:36:18 2022] DELETE /identity/v3/users/7274fc324ce044c2b35541621fc5ec7d => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:18.706251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48171e53-f2f9-4ee9-8315-d9c499dd50b6 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:18.706744 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48171e53-f2f9-4ee9-8315-d9c499dd50b6 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:18.706873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48171e53-f2f9-4ee9-8315-d9c499dd50b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:18.728147 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-48171e53-f2f9-4ee9-8315-d9c499dd50b6 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:18.729425 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-48171e53-f2f9-4ee9-8315-d9c499dd50b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:18.791427 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1161/2320] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:18.801938 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c583e5-f89d-4a27-a1d7-410fab1fa2b8 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:18.803364 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c1c583e5-f89d-4a27-a1d7-410fab1fa2b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:18.894983 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c583e5-f89d-4a27-a1d7-410fab1fa2b8 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:18.898905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e350faae-6401-47a3-beec-c34d34901aa5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:18.899382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e350faae-6401-47a3-beec-c34d34901aa5 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:18.899655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e350faae-6401-47a3-beec-c34d34901aa5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:18.899926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e350faae-6401-47a3-beec-c34d34901aa5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:18.910620 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e350faae-6401-47a3-beec-c34d34901aa5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:18.911958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e350faae-6401-47a3-beec-c34d34901aa5 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:18.924087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1160/2321] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:36:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:36:19.273489 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48171e53-f2f9-4ee9-8315-d9c499dd50b6 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:19.275029 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-48171e53-f2f9-4ee9-8315-d9c499dd50b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:19.289560 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48171e53-f2f9-4ee9-8315-d9c499dd50b6 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:19.295271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282fbb78-5f0d-4ddb-9466-936918e48499 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:19.295905 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282fbb78-5f0d-4ddb-9466-936918e48499 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:19.296177 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282fbb78-5f0d-4ddb-9466-936918e48499 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:19.296485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282fbb78-5f0d-4ddb-9466-936918e48499 admin admin] PATH_INFO: `/v3/projects/1e77d61e02534817a1a37b83a01a0fb4` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:19.297630 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282fbb78-5f0d-4ddb-9466-936918e48499 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1e77d61e02534817a1a37b83a01a0fb4)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:19.300594 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282fbb78-5f0d-4ddb-9466-936918e48499 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:19.355400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-282fbb78-5f0d-4ddb-9466-936918e48499 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1e77d61e02534817a1a37b83a01a0fb4', 'request_id': 'req-282fbb78-5f0d-4ddb-9466-936918e48499'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:19.365753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-282fbb78-5f0d-4ddb-9466-936918e48499 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1e77d61e02534817a1a37b83a01a0fb4', 'request_id': 'req-282fbb78-5f0d-4ddb-9466-936918e48499'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:19.372508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-282fbb78-5f0d-4ddb-9466-936918e48499 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1e77d61e02534817a1a37b83a01a0fb4', 'request_id': 'req-282fbb78-5f0d-4ddb-9466-936918e48499'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:19.386098 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-282fbb78-5f0d-4ddb-9466-936918e48499 admin admin] The token cache is being invalidate because project 1e77d61e02534817a1a37b83a01a0fb4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:36:19.386394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-282fbb78-5f0d-4ddb-9466-936918e48499 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:19.387172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-282fbb78-5f0d-4ddb-9466-936918e48499 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:19.388807 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1162/2322] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:36:19 2022] DELETE /identity/v3/projects/1e77d61e02534817a1a37b83a01a0fb4 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:19.394128 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e350faae-6401-47a3-beec-c34d34901aa5 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:19.399166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:19.408695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e350faae-6401-47a3-beec-c34d34901aa5 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:19.483033 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfdea8d2-40cd-4bc7-9368-5a30b6cf11eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:19.483729 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdea8d2-40cd-4bc7-9368-5a30b6cf11eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:19.484032 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdea8d2-40cd-4bc7-9368-5a30b6cf11eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:19.484334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdea8d2-40cd-4bc7-9368-5a30b6cf11eb admin admin] PATH_INFO: `/v3/users/7a24d295879847219a68d38ee5b59633` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:19.485589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfdea8d2-40cd-4bc7-9368-5a30b6cf11eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=7a24d295879847219a68d38ee5b59633)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:19.487528 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfdea8d2-40cd-4bc7-9368-5a30b6cf11eb admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:19.538501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cfdea8d2-40cd-4bc7-9368-5a30b6cf11eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7a24d295879847219a68d38ee5b59633'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:19.548191 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cfdea8d2-40cd-4bc7-9368-5a30b6cf11eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7a24d295879847219a68d38ee5b59633'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:19.557377 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cfdea8d2-40cd-4bc7-9368-5a30b6cf11eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7a24d295879847219a68d38ee5b59633'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:19.570712 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cfdea8d2-40cd-4bc7-9368-5a30b6cf11eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7a24d295879847219a68d38ee5b59633'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:19.571502 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cfdea8d2-40cd-4bc7-9368-5a30b6cf11eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7a24d295879847219a68d38ee5b59633'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:19.578366 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1161/2323] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:36:19 2022] DELETE /identity/v3/users/7a24d295879847219a68d38ee5b59633 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:19.883052 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282fbb78-5f0d-4ddb-9466-936918e48499 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:19.884754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-282fbb78-5f0d-4ddb-9466-936918e48499 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:19.902158 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282fbb78-5f0d-4ddb-9466-936918e48499 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:19.970208 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110beaf4-817f-4fb3-b361-a17a820ca9c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:19.971260 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110beaf4-817f-4fb3-b361-a17a820ca9c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:19.971484 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110beaf4-817f-4fb3-b361-a17a820ca9c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:19.971765 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110beaf4-817f-4fb3-b361-a17a820ca9c5 admin admin] PATH_INFO: `/v3/projects/b7f36be12d394ea6abafee8630aed906` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:19.973056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110beaf4-817f-4fb3-b361-a17a820ca9c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b7f36be12d394ea6abafee8630aed906)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:19.975038 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110beaf4-817f-4fb3-b361-a17a820ca9c5 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:20.012646 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:20.041852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-110beaf4-817f-4fb3-b361-a17a820ca9c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b7f36be12d394ea6abafee8630aed906', 'request_id': 'req-110beaf4-817f-4fb3-b361-a17a820ca9c5'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:20.053203 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-110beaf4-817f-4fb3-b361-a17a820ca9c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b7f36be12d394ea6abafee8630aed906', 'request_id': 'req-110beaf4-817f-4fb3-b361-a17a820ca9c5'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:20.071343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-110beaf4-817f-4fb3-b361-a17a820ca9c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b7f36be12d394ea6abafee8630aed906', 'request_id': 'req-110beaf4-817f-4fb3-b361-a17a820ca9c5'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:20.091221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-110beaf4-817f-4fb3-b361-a17a820ca9c5 admin admin] The token cache is being invalidate because project b7f36be12d394ea6abafee8630aed906 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:36:20.092199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-110beaf4-817f-4fb3-b361-a17a820ca9c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:20.092199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-110beaf4-817f-4fb3-b361-a17a820ca9c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:20.093797 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1163/2324] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:36:19 2022] DELETE /identity/v3/projects/b7f36be12d394ea6abafee8630aed906 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:20.099878 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c8bdf0-feab-41b2-b762-93d9505916a9 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:20.100680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c8bdf0-feab-41b2-b762-93d9505916a9 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:20.101152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c8bdf0-feab-41b2-b762-93d9505916a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:20.124090 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-26c8bdf0-feab-41b2-b762-93d9505916a9 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:20.125356 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-26c8bdf0-feab-41b2-b762-93d9505916a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:20.197825 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1162/2325] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:20.207157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb089497-941a-4f2f-9f69-ce03b238c977 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:20.207443 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb089497-941a-4f2f-9f69-ce03b238c977 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:20.207723 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb089497-941a-4f2f-9f69-ce03b238c977 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:20.229024 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-fb089497-941a-4f2f-9f69-ce03b238c977 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:20.231819 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-fb089497-941a-4f2f-9f69-ce03b238c977 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:20.240020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1164/2326] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:20.244451 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c8bdf0-feab-41b2-b762-93d9505916a9 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:20.257399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c8bdf0-feab-41b2-b762-93d9505916a9 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:20.266962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7053822-d7e7-4e6a-9db0-f602ee08128a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:20.268006 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7053822-d7e7-4e6a-9db0-f602ee08128a admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:20.268544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7053822-d7e7-4e6a-9db0-f602ee08128a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:20.269049 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7053822-d7e7-4e6a-9db0-f602ee08128a admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:20.270272 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7053822-d7e7-4e6a-9db0-f602ee08128a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:36:20.270811 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7053822-d7e7-4e6a-9db0-f602ee08128a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:20.273953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7053822-d7e7-4e6a-9db0-f602ee08128a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:20.298654 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1163/2327] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:36:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:36:20.303252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb089497-941a-4f2f-9f69-ce03b238c977 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:20.313630 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb089497-941a-4f2f-9f69-ce03b238c977 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:20.318307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2759392d-93ae-4637-b102-8de49bf03659 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:20.318907 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2759392d-93ae-4637-b102-8de49bf03659 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:20.319187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2759392d-93ae-4637-b102-8de49bf03659 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:20.319461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2759392d-93ae-4637-b102-8de49bf03659 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:20.320092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2759392d-93ae-4637-b102-8de49bf03659 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:20.322081 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2759392d-93ae-4637-b102-8de49bf03659 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:20.352580 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1165/2328] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:36:20 2022] POST /identity/v3/projects => generated 383 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:36:20.359491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7053822-d7e7-4e6a-9db0-f602ee08128a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:20.370870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7053822-d7e7-4e6a-9db0-f602ee08128a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:20.376771 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:20.444861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c70eba-80de-4b40-ae8e-ebcf0c569a3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:20.445479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c70eba-80de-4b40-ae8e-ebcf0c569a3d admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:20.445766 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c70eba-80de-4b40-ae8e-ebcf0c569a3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:20.446042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c70eba-80de-4b40-ae8e-ebcf0c569a3d admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:20.446699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c70eba-80de-4b40-ae8e-ebcf0c569a3d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:20.449249 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c70eba-80de-4b40-ae8e-ebcf0c569a3d admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:20.472563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1164/2329] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:36:20 2022] POST /identity/v3/users => generated 292 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:36:20.477133 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2759392d-93ae-4637-b102-8de49bf03659 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:20.488205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2759392d-93ae-4637-b102-8de49bf03659 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:20.493050 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86d84e3-6da0-4208-a2aa-7bec4b0c924c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:20.493665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86d84e3-6da0-4208-a2aa-7bec4b0c924c admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:20.493950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86d84e3-6da0-4208-a2aa-7bec4b0c924c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:20.494223 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86d84e3-6da0-4208-a2aa-7bec4b0c924c admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:20.494869 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86d84e3-6da0-4208-a2aa-7bec4b0c924c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:20.496732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86d84e3-6da0-4208-a2aa-7bec4b0c924c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:20.508647 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1166/2330] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:36:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:36:20.512895 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c70eba-80de-4b40-ae8e-ebcf0c569a3d admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:20.523761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c70eba-80de-4b40-ae8e-ebcf0c569a3d admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:20.529359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c20e2b-6820-4937-ad0e-c626281ea290 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:20.529978 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c20e2b-6820-4937-ad0e-c626281ea290 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:20.530523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c20e2b-6820-4937-ad0e-c626281ea290 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:20.530792 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c20e2b-6820-4937-ad0e-c626281ea290 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:20.531444 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c20e2b-6820-4937-ad0e-c626281ea290 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:20.533101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c20e2b-6820-4937-ad0e-c626281ea290 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:20.549614 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1165/2331] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:36:20 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:36:20.551023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86d84e3-6da0-4208-a2aa-7bec4b0c924c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:20.568863 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86d84e3-6da0-4208-a2aa-7bec4b0c924c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:20.574791 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be65e303-ef04-4715-ba06-d1be56081331 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:20.575637 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be65e303-ef04-4715-ba06-d1be56081331 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:20.575878 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be65e303-ef04-4715-ba06-d1be56081331 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:20.576164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be65e303-ef04-4715-ba06-d1be56081331 admin admin] PATH_INFO: `/v3/projects/97ea890ae27d4d238690ff1e8fe3bffe/users/b76323dfcec7458aaa992835b80aa680/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:20.595483 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be65e303-ef04-4715-ba06-d1be56081331 admin admin] RBAC: Authorizing `identity:create_grant(project_id=97ea890ae27d4d238690ff1e8fe3bffe, user_id=b76323dfcec7458aaa992835b80aa680, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:20.598728 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be65e303-ef04-4715-ba06-d1be56081331 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:20.627291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1167/2332] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:36:20 2022] PUT /identity/v3/projects/97ea890ae27d4d238690ff1e8fe3bffe/users/b76323dfcec7458aaa992835b80aa680/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:36:20.629869 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c816dc0-6c1b-46d2-b0a5-7262127d4492 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:20.630338 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c816dc0-6c1b-46d2-b0a5-7262127d4492 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:20.630798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c816dc0-6c1b-46d2-b0a5-7262127d4492 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:20.667973 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-6c816dc0-6c1b-46d2-b0a5-7262127d4492 None None] MFA Rules not processed for user `b76323dfcec7458aaa992835b80aa680`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:20.667973 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6c816dc0-6c1b-46d2-b0a5-7262127d4492 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:20.747807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1166/2333] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:20 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:20.752651 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be65e303-ef04-4715-ba06-d1be56081331 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:20.829296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be65e303-ef04-4715-ba06-d1be56081331 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:20.836108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b646bba-0ae6-4ede-867f-1170c136bc61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:20.837217 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b646bba-0ae6-4ede-867f-1170c136bc61 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:20.837740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b646bba-0ae6-4ede-867f-1170c136bc61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:20.838245 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b646bba-0ae6-4ede-867f-1170c136bc61 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:20.839601 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b646bba-0ae6-4ede-867f-1170c136bc61 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:20.843443 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b646bba-0ae6-4ede-867f-1170c136bc61 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:20.877926 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1168/2334] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:36:20 2022] POST /identity/v3/projects => generated 381 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:36:20.881915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c816dc0-6c1b-46d2-b0a5-7262127d4492 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:20.897141 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c816dc0-6c1b-46d2-b0a5-7262127d4492 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:20.983173 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5c93dc-472e-4f90-8193-7e4cbceac89b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:20.983789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5c93dc-472e-4f90-8193-7e4cbceac89b admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:20.984066 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5c93dc-472e-4f90-8193-7e4cbceac89b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:20.984447 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5c93dc-472e-4f90-8193-7e4cbceac89b admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:20.985099 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5c93dc-472e-4f90-8193-7e4cbceac89b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:20.987184 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5c93dc-472e-4f90-8193-7e4cbceac89b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:21.020164 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1167/2335] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:36:20 2022] POST /identity/v3/users => generated 297 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:36:21.022082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b646bba-0ae6-4ede-867f-1170c136bc61 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:21.033948 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b646bba-0ae6-4ede-867f-1170c136bc61 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:21.039122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8862383c-88ba-4765-aedf-5dbc12fc5a3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:21.039705 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8862383c-88ba-4765-aedf-5dbc12fc5a3d admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:21.040058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8862383c-88ba-4765-aedf-5dbc12fc5a3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:21.040460 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8862383c-88ba-4765-aedf-5dbc12fc5a3d admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:21.041213 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8862383c-88ba-4765-aedf-5dbc12fc5a3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:21.043151 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8862383c-88ba-4765-aedf-5dbc12fc5a3d admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:21.055327 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1169/2336] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:36:21 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:36:21.061097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5c93dc-472e-4f90-8193-7e4cbceac89b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:21.071862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5c93dc-472e-4f90-8193-7e4cbceac89b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:21.078415 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8898489-5b5b-409c-865c-d54f0009d746 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:21.079124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8898489-5b5b-409c-865c-d54f0009d746 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:21.079474 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8898489-5b5b-409c-865c-d54f0009d746 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:21.079677 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8898489-5b5b-409c-865c-d54f0009d746 admin admin] PATH_INFO: `/v3/projects/446b288af4fb46ecb0703887cc97a6cb/users/c420d8b18e5749e685801d03772523c5/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:21.082558 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:21.099120 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8898489-5b5b-409c-865c-d54f0009d746 admin admin] RBAC: Authorizing `identity:create_grant(project_id=446b288af4fb46ecb0703887cc97a6cb, user_id=c420d8b18e5749e685801d03772523c5, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:21.102157 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8898489-5b5b-409c-865c-d54f0009d746 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:21.118861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1168/2337] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:36:21 2022] PUT /identity/v3/projects/446b288af4fb46ecb0703887cc97a6cb/users/c420d8b18e5749e685801d03772523c5/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:36:21.123104 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8862383c-88ba-4765-aedf-5dbc12fc5a3d admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:21.136634 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8862383c-88ba-4765-aedf-5dbc12fc5a3d admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:21.161047 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:21.208526 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b033ff1-c570-45a8-83ea-fb5c385ff50a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:21.209003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b033ff1-c570-45a8-83ea-fb5c385ff50a admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:21.209288 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b033ff1-c570-45a8-83ea-fb5c385ff50a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:21.209558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b033ff1-c570-45a8-83ea-fb5c385ff50a admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:21.210231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b033ff1-c570-45a8-83ea-fb5c385ff50a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:21.211854 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b033ff1-c570-45a8-83ea-fb5c385ff50a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:21.223823 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1170/2338] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:36:21 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:36:21.228005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8898489-5b5b-409c-865c-d54f0009d746 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:21.239000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8898489-5b5b-409c-865c-d54f0009d746 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:21.243932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901221cb-207e-44b3-a6dc-d2882a783758 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:21.244638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901221cb-207e-44b3-a6dc-d2882a783758 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:21.245219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901221cb-207e-44b3-a6dc-d2882a783758 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:21.245219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901221cb-207e-44b3-a6dc-d2882a783758 admin admin] PATH_INFO: `/v3/projects/446b288af4fb46ecb0703887cc97a6cb/users/c420d8b18e5749e685801d03772523c5/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:21.247256 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901221cb-207e-44b3-a6dc-d2882a783758 admin admin] RBAC: Authorizing `identity:create_grant(project_id=446b288af4fb46ecb0703887cc97a6cb, user_id=c420d8b18e5749e685801d03772523c5, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:21.250021 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901221cb-207e-44b3-a6dc-d2882a783758 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:21.262517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1169/2339] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:36:21 2022] PUT /identity/v3/projects/446b288af4fb46ecb0703887cc97a6cb/users/c420d8b18e5749e685801d03772523c5/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:21.272721 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96426760-6c7a-473e-93f3-0f2a0289f81f None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:21.273018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96426760-6c7a-473e-93f3-0f2a0289f81f None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:21.273302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96426760-6c7a-473e-93f3-0f2a0289f81f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:21.309063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-96426760-6c7a-473e-93f3-0f2a0289f81f None None] MFA Rules not processed for user `c420d8b18e5749e685801d03772523c5`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:21.310297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-96426760-6c7a-473e-93f3-0f2a0289f81f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:21.403251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1171/2340] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:21 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:21.415123 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901221cb-207e-44b3-a6dc-d2882a783758 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:21.417045 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-901221cb-207e-44b3-a6dc-d2882a783758 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:21.488784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901221cb-207e-44b3-a6dc-d2882a783758 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:21.492437 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb162ac7-7dc4-4754-82cb-92a2954ad03c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:21.493046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb162ac7-7dc4-4754-82cb-92a2954ad03c service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:21.493849 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb162ac7-7dc4-4754-82cb-92a2954ad03c service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:21.493849 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb162ac7-7dc4-4754-82cb-92a2954ad03c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:21.503649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb162ac7-7dc4-4754-82cb-92a2954ad03c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:21.505122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb162ac7-7dc4-4754-82cb-92a2954ad03c service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:21.516079 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1170/2341] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:36:21 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:36:21.630295 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96426760-6c7a-473e-93f3-0f2a0289f81f None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:21.641002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96426760-6c7a-473e-93f3-0f2a0289f81f None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:21.644916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae844808-1a3e-444e-a68e-316c317b10f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:21.645588 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae844808-1a3e-444e-a68e-316c317b10f6 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:21.645862 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae844808-1a3e-444e-a68e-316c317b10f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:21.646130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae844808-1a3e-444e-a68e-316c317b10f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:21.714863 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae844808-1a3e-444e-a68e-316c317b10f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:21.716846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae844808-1a3e-444e-a68e-316c317b10f6 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:21.733286 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1172/2342] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:36:21 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:36:24.610186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb162ac7-7dc4-4754-82cb-92a2954ad03c service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:24.678347 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb162ac7-7dc4-4754-82cb-92a2954ad03c service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:24.683034 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97dc34c-840e-4a1b-8720-cb0b3e6b79e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:24.683677 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97dc34c-840e-4a1b-8720-cb0b3e6b79e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:24.683948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97dc34c-840e-4a1b-8720-cb0b3e6b79e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:24.684221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97dc34c-840e-4a1b-8720-cb0b3e6b79e3 admin admin] PATH_INFO: `/v3/users/b76323dfcec7458aaa992835b80aa680` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:24.685851 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97dc34c-840e-4a1b-8720-cb0b3e6b79e3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b76323dfcec7458aaa992835b80aa680)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:24.687788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97dc34c-840e-4a1b-8720-cb0b3e6b79e3 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:24.743447 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-d97dc34c-840e-4a1b-8720-cb0b3e6b79e3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b76323dfcec7458aaa992835b80aa680'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:24.755623 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-d97dc34c-840e-4a1b-8720-cb0b3e6b79e3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b76323dfcec7458aaa992835b80aa680'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:24.765218 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-d97dc34c-840e-4a1b-8720-cb0b3e6b79e3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b76323dfcec7458aaa992835b80aa680'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:24.779588 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=88,keystone:INSERT=3,keystone:DELETE=9 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:24.787933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-d97dc34c-840e-4a1b-8720-cb0b3e6b79e3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b76323dfcec7458aaa992835b80aa680'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:24.789444 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-d97dc34c-840e-4a1b-8720-cb0b3e6b79e3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b76323dfcec7458aaa992835b80aa680'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:24.800765 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1171/2343] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:36:24 2022] DELETE /identity/v3/users/b76323dfcec7458aaa992835b80aa680 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:24.806874 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461946aa-cabf-46cd-8afb-0b2ead61291e None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:24.807446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461946aa-cabf-46cd-8afb-0b2ead61291e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:24.807446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461946aa-cabf-46cd-8afb-0b2ead61291e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:24.828508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-461946aa-cabf-46cd-8afb-0b2ead61291e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:24.829743 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-461946aa-cabf-46cd-8afb-0b2ead61291e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:24.903590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:24.952302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1173/2344] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:24.959669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97dc34c-840e-4a1b-8720-cb0b3e6b79e3 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:24.961269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d97dc34c-840e-4a1b-8720-cb0b3e6b79e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:25.049275 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97dc34c-840e-4a1b-8720-cb0b3e6b79e3 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:25.053184 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4af4e0-2698-405b-8ad8-23526c51c6b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:25.053804 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4af4e0-2698-405b-8ad8-23526c51c6b8 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:25.054080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4af4e0-2698-405b-8ad8-23526c51c6b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:25.054349 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4af4e0-2698-405b-8ad8-23526c51c6b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:25.065409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4af4e0-2698-405b-8ad8-23526c51c6b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:25.066908 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4af4e0-2698-405b-8ad8-23526c51c6b8 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:25.078239 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1172/2345] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:36:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:36:25.431428 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461946aa-cabf-46cd-8afb-0b2ead61291e None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:25.434458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-461946aa-cabf-46cd-8afb-0b2ead61291e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:25.473969 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461946aa-cabf-46cd-8afb-0b2ead61291e None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:25.498803 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6411c2-c89f-4321-be89-9494dad5144f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:25.499539 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6411c2-c89f-4321-be89-9494dad5144f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:25.499940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6411c2-c89f-4321-be89-9494dad5144f admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:25.500157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6411c2-c89f-4321-be89-9494dad5144f admin admin] PATH_INFO: `/v3/projects/97ea890ae27d4d238690ff1e8fe3bffe` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:25.506583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6411c2-c89f-4321-be89-9494dad5144f admin admin] RBAC: Authorizing `identity:delete_project(project_id=97ea890ae27d4d238690ff1e8fe3bffe)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:25.508643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6411c2-c89f-4321-be89-9494dad5144f admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:25.586757 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-ed6411c2-c89f-4321-be89-9494dad5144f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '97ea890ae27d4d238690ff1e8fe3bffe', 'request_id': 'req-ed6411c2-c89f-4321-be89-9494dad5144f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:25.596404 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-ed6411c2-c89f-4321-be89-9494dad5144f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '97ea890ae27d4d238690ff1e8fe3bffe', 'request_id': 'req-ed6411c2-c89f-4321-be89-9494dad5144f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:25.606967 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-ed6411c2-c89f-4321-be89-9494dad5144f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '97ea890ae27d4d238690ff1e8fe3bffe', 'request_id': 'req-ed6411c2-c89f-4321-be89-9494dad5144f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:25.627662 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-ed6411c2-c89f-4321-be89-9494dad5144f admin admin] The token cache is being invalidate because project 97ea890ae27d4d238690ff1e8fe3bffe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:36:25.627961 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-ed6411c2-c89f-4321-be89-9494dad5144f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:25.628633 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-ed6411c2-c89f-4321-be89-9494dad5144f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:25.631321 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1174/2346] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:36:25 2022] DELETE /identity/v3/projects/97ea890ae27d4d238690ff1e8fe3bffe => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:25.637131 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4af4e0-2698-405b-8ad8-23526c51c6b8 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:25.651028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4af4e0-2698-405b-8ad8-23526c51c6b8 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:25.732093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afeaf7ee-b70a-41cd-8e60-08aafcfb8899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:25.733199 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afeaf7ee-b70a-41cd-8e60-08aafcfb8899 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:25.733677 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afeaf7ee-b70a-41cd-8e60-08aafcfb8899 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:25.734151 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afeaf7ee-b70a-41cd-8e60-08aafcfb8899 admin admin] PATH_INFO: `/v3/users/c420d8b18e5749e685801d03772523c5` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:25.736123 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afeaf7ee-b70a-41cd-8e60-08aafcfb8899 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c420d8b18e5749e685801d03772523c5)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:25.739583 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afeaf7ee-b70a-41cd-8e60-08aafcfb8899 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:25.817859 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-afeaf7ee-b70a-41cd-8e60-08aafcfb8899 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c420d8b18e5749e685801d03772523c5'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:25.828530 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-afeaf7ee-b70a-41cd-8e60-08aafcfb8899 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c420d8b18e5749e685801d03772523c5'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:25.841000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-afeaf7ee-b70a-41cd-8e60-08aafcfb8899 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c420d8b18e5749e685801d03772523c5'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:25.856561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-afeaf7ee-b70a-41cd-8e60-08aafcfb8899 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c420d8b18e5749e685801d03772523c5'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:25.857301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-afeaf7ee-b70a-41cd-8e60-08aafcfb8899 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c420d8b18e5749e685801d03772523c5'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:25.863635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1173/2347] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:36:25 2022] DELETE /identity/v3/users/c420d8b18e5749e685801d03772523c5 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:26.233870 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6411c2-c89f-4321-be89-9494dad5144f admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:26.237178 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ed6411c2-c89f-4321-be89-9494dad5144f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:26.273309 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6411c2-c89f-4321-be89-9494dad5144f admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:26.396869 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0697dcb8-7c24-44a0-ab28-d0619eca1389 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:26.397890 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0697dcb8-7c24-44a0-ab28-d0619eca1389 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:26.398359 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0697dcb8-7c24-44a0-ab28-d0619eca1389 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:26.398879 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0697dcb8-7c24-44a0-ab28-d0619eca1389 admin admin] PATH_INFO: `/v3/projects/446b288af4fb46ecb0703887cc97a6cb` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:26.400886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0697dcb8-7c24-44a0-ab28-d0619eca1389 admin admin] RBAC: Authorizing `identity:delete_project(project_id=446b288af4fb46ecb0703887cc97a6cb)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:26.405681 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0697dcb8-7c24-44a0-ab28-d0619eca1389 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:26.465527 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0697dcb8-7c24-44a0-ab28-d0619eca1389 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '446b288af4fb46ecb0703887cc97a6cb', 'request_id': 'req-0697dcb8-7c24-44a0-ab28-d0619eca1389'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:26.473426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0697dcb8-7c24-44a0-ab28-d0619eca1389 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '446b288af4fb46ecb0703887cc97a6cb', 'request_id': 'req-0697dcb8-7c24-44a0-ab28-d0619eca1389'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:26.482169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0697dcb8-7c24-44a0-ab28-d0619eca1389 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '446b288af4fb46ecb0703887cc97a6cb', 'request_id': 'req-0697dcb8-7c24-44a0-ab28-d0619eca1389'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:26.497358 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:26.504796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0697dcb8-7c24-44a0-ab28-d0619eca1389 admin admin] The token cache is being invalidate because project 446b288af4fb46ecb0703887cc97a6cb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:36:26.505410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0697dcb8-7c24-44a0-ab28-d0619eca1389 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:26.506840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0697dcb8-7c24-44a0-ab28-d0619eca1389 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:26.509507 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1175/2348] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:36:26 2022] DELETE /identity/v3/projects/446b288af4fb46ecb0703887cc97a6cb => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:26.516433 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e160e0-9f9e-4e25-969f-99fae900abbf None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:26.516841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e160e0-9f9e-4e25-969f-99fae900abbf None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:26.517217 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e160e0-9f9e-4e25-969f-99fae900abbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:26.541197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-c9e160e0-9f9e-4e25-969f-99fae900abbf None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:26.543888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c9e160e0-9f9e-4e25-969f-99fae900abbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:26.555866 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:26.689299 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1174/2349] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:26.700187 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae742f3-9449-4ad0-93ef-e2aa4453ff55 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:26.700772 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae742f3-9449-4ad0-93ef-e2aa4453ff55 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:26.701177 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae742f3-9449-4ad0-93ef-e2aa4453ff55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:26.723228 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-4ae742f3-9449-4ad0-93ef-e2aa4453ff55 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:26.724864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-4ae742f3-9449-4ad0-93ef-e2aa4453ff55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:26.738510 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1176/2350] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:36:26.742964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e160e0-9f9e-4e25-969f-99fae900abbf None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:26.754230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e160e0-9f9e-4e25-969f-99fae900abbf None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:26.759480 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4400324f-666e-4ffa-8ced-1a4f067b74db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:26.760229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4400324f-666e-4ffa-8ced-1a4f067b74db admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:26.760691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4400324f-666e-4ffa-8ced-1a4f067b74db admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:26.761478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4400324f-666e-4ffa-8ced-1a4f067b74db admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:26.762331 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4400324f-666e-4ffa-8ced-1a4f067b74db admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:36:26.762645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4400324f-666e-4ffa-8ced-1a4f067b74db admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:26.764352 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4400324f-666e-4ffa-8ced-1a4f067b74db admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:26.778867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1175/2351] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:36:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:36:26.785485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae742f3-9449-4ad0-93ef-e2aa4453ff55 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:26.797367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae742f3-9449-4ad0-93ef-e2aa4453ff55 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:26.802945 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95bf5b4-89d4-4ffb-9f9d-fda07338e251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:26.803567 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95bf5b4-89d4-4ffb-9f9d-fda07338e251 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:26.803927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95bf5b4-89d4-4ffb-9f9d-fda07338e251 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:26.804204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95bf5b4-89d4-4ffb-9f9d-fda07338e251 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:26.804985 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95bf5b4-89d4-4ffb-9f9d-fda07338e251 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:26.807227 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95bf5b4-89d4-4ffb-9f9d-fda07338e251 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:26.843497 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1177/2352] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:36:26 2022] POST /identity/v3/projects => generated 383 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:36:26.847725 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4400324f-666e-4ffa-8ced-1a4f067b74db admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:26.859792 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4400324f-666e-4ffa-8ced-1a4f067b74db admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:26.924267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db9fe56-dab7-446f-925c-8227376ea85b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:26.924817 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db9fe56-dab7-446f-925c-8227376ea85b admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:26.925095 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db9fe56-dab7-446f-925c-8227376ea85b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:26.925365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db9fe56-dab7-446f-925c-8227376ea85b admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:26.926013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db9fe56-dab7-446f-925c-8227376ea85b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:26.927959 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db9fe56-dab7-446f-925c-8227376ea85b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:26.951452 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1176/2353] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:36:26 2022] POST /identity/v3/users => generated 292 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:36:26.955717 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95bf5b4-89d4-4ffb-9f9d-fda07338e251 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:26.968772 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95bf5b4-89d4-4ffb-9f9d-fda07338e251 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:26.975950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbda354d-7004-4aba-9184-500f4a8c086a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:26.977026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbda354d-7004-4aba-9184-500f4a8c086a admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:26.977409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbda354d-7004-4aba-9184-500f4a8c086a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:26.978281 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbda354d-7004-4aba-9184-500f4a8c086a admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:26.979157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbda354d-7004-4aba-9184-500f4a8c086a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:26.981470 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbda354d-7004-4aba-9184-500f4a8c086a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:26.996978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1178/2354] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:36:26 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:36:27.001208 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db9fe56-dab7-446f-925c-8227376ea85b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:27.012946 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db9fe56-dab7-446f-925c-8227376ea85b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:27.019526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b117669f-370a-4218-b8ef-b84571a918d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:27.020525 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b117669f-370a-4218-b8ef-b84571a918d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:27.020901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b117669f-370a-4218-b8ef-b84571a918d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:27.021284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b117669f-370a-4218-b8ef-b84571a918d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:27.022074 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b117669f-370a-4218-b8ef-b84571a918d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:27.023800 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b117669f-370a-4218-b8ef-b84571a918d7 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:27.037938 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1177/2355] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:36:26 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:36:27.043241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbda354d-7004-4aba-9184-500f4a8c086a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:27.054394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbda354d-7004-4aba-9184-500f4a8c086a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:27.065083 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c180bec3-9ebf-4880-87a9-7941587acc27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:27.065805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c180bec3-9ebf-4880-87a9-7941587acc27 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:27.066084 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c180bec3-9ebf-4880-87a9-7941587acc27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:27.066357 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c180bec3-9ebf-4880-87a9-7941587acc27 admin admin] PATH_INFO: `/v3/projects/006605db295a44dab622e7b7f2dcfe49/users/bfb24b9fa3b24e438b28144e73cce45a/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:27.085511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c180bec3-9ebf-4880-87a9-7941587acc27 admin admin] RBAC: Authorizing `identity:create_grant(project_id=006605db295a44dab622e7b7f2dcfe49, user_id=bfb24b9fa3b24e438b28144e73cce45a, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:27.091384 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c180bec3-9ebf-4880-87a9-7941587acc27 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:27.114113 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1179/2356] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:36:27 2022] PUT /identity/v3/projects/006605db295a44dab622e7b7f2dcfe49/users/bfb24b9fa3b24e438b28144e73cce45a/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:36:27.127216 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7237417e-ca34-4a06-9970-7e2b35ce3c15 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:27.127709 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7237417e-ca34-4a06-9970-7e2b35ce3c15 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:27.128187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7237417e-ca34-4a06-9970-7e2b35ce3c15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:27.172083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-7237417e-ca34-4a06-9970-7e2b35ce3c15 None None] MFA Rules not processed for user `bfb24b9fa3b24e438b28144e73cce45a`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:27.173759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7237417e-ca34-4a06-9970-7e2b35ce3c15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:27.257443 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1178/2357] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:27 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:27.270439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c180bec3-9ebf-4880-87a9-7941587acc27 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:27.272271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c180bec3-9ebf-4880-87a9-7941587acc27 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:27.343404 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c180bec3-9ebf-4880-87a9-7941587acc27 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:27.350022 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6b9f94-a495-474d-8ede-5d61721e09bf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:27.350977 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6b9f94-a495-474d-8ede-5d61721e09bf service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:27.351375 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6b9f94-a495-474d-8ede-5d61721e09bf service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:27.351524 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6b9f94-a495-474d-8ede-5d61721e09bf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:27.366599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6b9f94-a495-474d-8ede-5d61721e09bf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:27.366599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6b9f94-a495-474d-8ede-5d61721e09bf service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:27.377618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1180/2358] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:36:27 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:36:27.407025 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7237417e-ca34-4a06-9970-7e2b35ce3c15 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:27.419745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:27.533254 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7237417e-ca34-4a06-9970-7e2b35ce3c15 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:27.541154 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f8322b-d4ca-4766-96fa-6b3f03086d8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:27.542769 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f8322b-d4ca-4766-96fa-6b3f03086d8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:27.543207 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f8322b-d4ca-4766-96fa-6b3f03086d8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:27.543732 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f8322b-d4ca-4766-96fa-6b3f03086d8a admin admin] PATH_INFO: `/v3/users/bfb24b9fa3b24e438b28144e73cce45a` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:27.546534 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f8322b-d4ca-4766-96fa-6b3f03086d8a admin admin] RBAC: Authorizing `identity:delete_user(user_id=bfb24b9fa3b24e438b28144e73cce45a)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:27.548920 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f8322b-d4ca-4766-96fa-6b3f03086d8a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:27.613530 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-17f8322b-d4ca-4766-96fa-6b3f03086d8a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bfb24b9fa3b24e438b28144e73cce45a'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:27.626877 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-17f8322b-d4ca-4766-96fa-6b3f03086d8a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bfb24b9fa3b24e438b28144e73cce45a'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:27.638411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-17f8322b-d4ca-4766-96fa-6b3f03086d8a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bfb24b9fa3b24e438b28144e73cce45a'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:27.663461 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-17f8322b-d4ca-4766-96fa-6b3f03086d8a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bfb24b9fa3b24e438b28144e73cce45a'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:27.664410 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-17f8322b-d4ca-4766-96fa-6b3f03086d8a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bfb24b9fa3b24e438b28144e73cce45a'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:27.673433 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1179/2359] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:36:27 2022] DELETE /identity/v3/users/bfb24b9fa3b24e438b28144e73cce45a => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:27.679651 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3b3b56-2a1c-4901-992c-76c2b6ba3beb None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:27.680298 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3b3b56-2a1c-4901-992c-76c2b6ba3beb None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:27.680808 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3b3b56-2a1c-4901-992c-76c2b6ba3beb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:27.708933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-bd3b3b56-2a1c-4901-992c-76c2b6ba3beb None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:27.710306 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-bd3b3b56-2a1c-4901-992c-76c2b6ba3beb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:27.822135 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1181/2360] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:36:27.860104 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f8322b-d4ca-4766-96fa-6b3f03086d8a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:27.861651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-17f8322b-d4ca-4766-96fa-6b3f03086d8a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:27.963495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f8322b-d4ca-4766-96fa-6b3f03086d8a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:27.970751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2eaf063-240c-410c-b5f2-98c21f02c43f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:27.971759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2eaf063-240c-410c-b5f2-98c21f02c43f service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:27.972251 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2eaf063-240c-410c-b5f2-98c21f02c43f service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:27.972680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2eaf063-240c-410c-b5f2-98c21f02c43f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:27.988898 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2eaf063-240c-410c-b5f2-98c21f02c43f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:27.992828 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2eaf063-240c-410c-b5f2-98c21f02c43f service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:28.015876 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1180/2361] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:36:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:36:28.605007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3b3b56-2a1c-4901-992c-76c2b6ba3beb None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:28.612027 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-bd3b3b56-2a1c-4901-992c-76c2b6ba3beb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:28.641421 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3b3b56-2a1c-4901-992c-76c2b6ba3beb None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:28.647452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b781e17-de2d-4bd8-a8aa-c45fb1abb394 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:28.649394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b781e17-de2d-4bd8-a8aa-c45fb1abb394 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:28.649394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b781e17-de2d-4bd8-a8aa-c45fb1abb394 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:28.649394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b781e17-de2d-4bd8-a8aa-c45fb1abb394 admin admin] PATH_INFO: `/v3/projects/006605db295a44dab622e7b7f2dcfe49` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:28.651246 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b781e17-de2d-4bd8-a8aa-c45fb1abb394 admin admin] RBAC: Authorizing `identity:delete_project(project_id=006605db295a44dab622e7b7f2dcfe49)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:28.653836 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b781e17-de2d-4bd8-a8aa-c45fb1abb394 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:28.719819 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:28.730712 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0b781e17-de2d-4bd8-a8aa-c45fb1abb394 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '006605db295a44dab622e7b7f2dcfe49', 'request_id': 'req-0b781e17-de2d-4bd8-a8aa-c45fb1abb394'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:28.737978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0b781e17-de2d-4bd8-a8aa-c45fb1abb394 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '006605db295a44dab622e7b7f2dcfe49', 'request_id': 'req-0b781e17-de2d-4bd8-a8aa-c45fb1abb394'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:28.746929 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0b781e17-de2d-4bd8-a8aa-c45fb1abb394 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '006605db295a44dab622e7b7f2dcfe49', 'request_id': 'req-0b781e17-de2d-4bd8-a8aa-c45fb1abb394'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:28.767513 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0b781e17-de2d-4bd8-a8aa-c45fb1abb394 admin admin] The token cache is being invalidate because project 006605db295a44dab622e7b7f2dcfe49 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:36:28.767804 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0b781e17-de2d-4bd8-a8aa-c45fb1abb394 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:28.768725 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-0b781e17-de2d-4bd8-a8aa-c45fb1abb394 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:28.770300 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1182/2362] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:36:28 2022] DELETE /identity/v3/projects/006605db295a44dab622e7b7f2dcfe49 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:28.775810 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecc40b7-57a3-4e60-b2a0-770721727fbe None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:28.776646 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecc40b7-57a3-4e60-b2a0-770721727fbe None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:28.776995 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecc40b7-57a3-4e60-b2a0-770721727fbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:28.803130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-fecc40b7-57a3-4e60-b2a0-770721727fbe None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:28.804379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-fecc40b7-57a3-4e60-b2a0-770721727fbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:28.839593 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:28.886903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1181/2363] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:28.895636 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cfefd6-1a30-4f38-ac5d-9a960f7d2668 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:28.895916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cfefd6-1a30-4f38-ac5d-9a960f7d2668 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:28.896209 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cfefd6-1a30-4f38-ac5d-9a960f7d2668 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:28.921018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-d5cfefd6-1a30-4f38-ac5d-9a960f7d2668 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:28.922785 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d5cfefd6-1a30-4f38-ac5d-9a960f7d2668 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:28.934437 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1183/2364] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:28.938985 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecc40b7-57a3-4e60-b2a0-770721727fbe None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:28.951953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecc40b7-57a3-4e60-b2a0-770721727fbe None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:28.960072 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2561ee69-ed40-4353-9492-e52b11fc6e8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:28.962136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2561ee69-ed40-4353-9492-e52b11fc6e8b admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:28.962522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2561ee69-ed40-4353-9492-e52b11fc6e8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:28.962951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2561ee69-ed40-4353-9492-e52b11fc6e8b admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:28.964077 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2561ee69-ed40-4353-9492-e52b11fc6e8b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:36:28.965268 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2561ee69-ed40-4353-9492-e52b11fc6e8b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:28.967516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2561ee69-ed40-4353-9492-e52b11fc6e8b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:28.991320 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1182/2365] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:36:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:36:28.992863 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cfefd6-1a30-4f38-ac5d-9a960f7d2668 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:29.018427 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cfefd6-1a30-4f38-ac5d-9a960f7d2668 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:29.025706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175ac93c-ab54-4188-a48b-93b0fb8008d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:29.026903 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175ac93c-ab54-4188-a48b-93b0fb8008d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:29.026903 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175ac93c-ab54-4188-a48b-93b0fb8008d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:29.027373 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175ac93c-ab54-4188-a48b-93b0fb8008d7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:29.027683 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175ac93c-ab54-4188-a48b-93b0fb8008d7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:29.031080 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175ac93c-ab54-4188-a48b-93b0fb8008d7 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:29.073832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1184/2366] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:36:28 2022] POST /identity/v3/projects => generated 383 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:36:29.078154 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2561ee69-ed40-4353-9492-e52b11fc6e8b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:29.092476 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2561ee69-ed40-4353-9492-e52b11fc6e8b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:29.176121 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5f0df2-9f4e-4a59-9b1d-e01c1a3955fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:29.181284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5f0df2-9f4e-4a59-9b1d-e01c1a3955fc admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:29.181601 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5f0df2-9f4e-4a59-9b1d-e01c1a3955fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:29.181874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5f0df2-9f4e-4a59-9b1d-e01c1a3955fc admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:29.183221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5f0df2-9f4e-4a59-9b1d-e01c1a3955fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:29.185538 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5f0df2-9f4e-4a59-9b1d-e01c1a3955fc admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:29.221303 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1183/2367] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:36:29 2022] POST /identity/v3/users => generated 292 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:36:29.222154 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175ac93c-ab54-4188-a48b-93b0fb8008d7 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:29.237741 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175ac93c-ab54-4188-a48b-93b0fb8008d7 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:29.244512 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ca74aa-996b-4413-a42e-7e4781cc4db3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:29.247135 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca74aa-996b-4413-a42e-7e4781cc4db3 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:29.247601 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca74aa-996b-4413-a42e-7e4781cc4db3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:29.248103 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca74aa-996b-4413-a42e-7e4781cc4db3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:29.256107 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ca74aa-996b-4413-a42e-7e4781cc4db3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:29.258082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ca74aa-996b-4413-a42e-7e4781cc4db3 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:29.271045 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1185/2368] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:36:29 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:36:29.278606 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5f0df2-9f4e-4a59-9b1d-e01c1a3955fc admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:29.299534 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5f0df2-9f4e-4a59-9b1d-e01c1a3955fc admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:29.308240 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c399d6-60ed-42e6-b688-a37fe1309339 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:29.309153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c399d6-60ed-42e6-b688-a37fe1309339 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:29.309699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c399d6-60ed-42e6-b688-a37fe1309339 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:29.310021 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c399d6-60ed-42e6-b688-a37fe1309339 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:29.310870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c399d6-60ed-42e6-b688-a37fe1309339 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:29.312833 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c399d6-60ed-42e6-b688-a37fe1309339 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:29.327692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1184/2369] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:36:29 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:36:29.333170 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ca74aa-996b-4413-a42e-7e4781cc4db3 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:29.350743 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ca74aa-996b-4413-a42e-7e4781cc4db3 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:29.359065 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a8ff15-e7b8-4bbb-8314-0bf833a5f69d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:29.360024 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a8ff15-e7b8-4bbb-8314-0bf833a5f69d admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:29.360504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a8ff15-e7b8-4bbb-8314-0bf833a5f69d admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:29.360870 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a8ff15-e7b8-4bbb-8314-0bf833a5f69d admin admin] PATH_INFO: `/v3/projects/d376e1791911493aac6eb714bd2b3bcf/users/c4e8ecc72ee24f3b88339a5fee7f3ccc/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:29.390216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a8ff15-e7b8-4bbb-8314-0bf833a5f69d admin admin] RBAC: Authorizing `identity:create_grant(project_id=d376e1791911493aac6eb714bd2b3bcf, user_id=c4e8ecc72ee24f3b88339a5fee7f3ccc, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:29.394566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a8ff15-e7b8-4bbb-8314-0bf833a5f69d admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:29.432924 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1186/2370] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:36:29 2022] PUT /identity/v3/projects/d376e1791911493aac6eb714bd2b3bcf/users/c4e8ecc72ee24f3b88339a5fee7f3ccc/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:29.442217 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937a4de2-0e52-4167-b436-6d64e5a125f1 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:29.442512 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937a4de2-0e52-4167-b436-6d64e5a125f1 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:29.442771 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937a4de2-0e52-4167-b436-6d64e5a125f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:29.483356 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-937a4de2-0e52-4167-b436-6d64e5a125f1 None None] MFA Rules not processed for user `c4e8ecc72ee24f3b88339a5fee7f3ccc`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:29.484843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-937a4de2-0e52-4167-b436-6d64e5a125f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:29.584561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1185/2371] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:29 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:29.602672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a8ff15-e7b8-4bbb-8314-0bf833a5f69d admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:29.605439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-49a8ff15-e7b8-4bbb-8314-0bf833a5f69d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:29.703100 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a8ff15-e7b8-4bbb-8314-0bf833a5f69d admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:29.708555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718536cc-41da-432a-b7f3-895bf158b844 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:29.709234 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718536cc-41da-432a-b7f3-895bf158b844 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:29.709694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718536cc-41da-432a-b7f3-895bf158b844 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:29.710099 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718536cc-41da-432a-b7f3-895bf158b844 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:29.723400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718536cc-41da-432a-b7f3-895bf158b844 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:29.725446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718536cc-41da-432a-b7f3-895bf158b844 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:29.741018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1187/2372] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:36:29 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:36:31.086619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199a3102-1173-4f96-b563-72579c0a29d6 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:31.087170 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199a3102-1173-4f96-b563-72579c0a29d6 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:31.087566 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199a3102-1173-4f96-b563-72579c0a29d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:31.127010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-199a3102-1173-4f96-b563-72579c0a29d6 None None] MFA Rules not processed for user `34afa4aac10c4a018ca05a0ba34c940a`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:31.128267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-199a3102-1173-4f96-b563-72579c0a29d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:31.144019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:31.210583 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1186/2373] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Oct 4 14:36:31 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:31.271001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718536cc-41da-432a-b7f3-895bf158b844 service cinder] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:31.272383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-718536cc-41da-432a-b7f3-895bf158b844 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:31.388680 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:31.399499 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718536cc-41da-432a-b7f3-895bf158b844 service cinder] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:31.407683 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bef174-99e0-484d-abd8-86a94bedc477 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5980e353164c4873bcdd2af847e0a1f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:31.408670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bef174-99e0-484d-abd8-86a94bedc477 service swift] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:31.408944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bef174-99e0-484d-abd8-86a94bedc477 service swift] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:31.409451 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bef174-99e0-484d-abd8-86a94bedc477 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:31.429932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bef174-99e0-484d-abd8-86a94bedc477 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:31.434033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bef174-99e0-484d-abd8-86a94bedc477 service swift] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:31.455181 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1188/2374] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Oct 4 14:36:31 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 14:36:36.025271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199a3102-1173-4f96-b563-72579c0a29d6 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:36.026785 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-199a3102-1173-4f96-b563-72579c0a29d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:36.189562 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199a3102-1173-4f96-b563-72579c0a29d6 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:36.202796 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762b6442-d884-4ae5-be83-e3fc394dd716 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:36.203564 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762b6442-d884-4ae5-be83-e3fc394dd716 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:36.203932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762b6442-d884-4ae5-be83-e3fc394dd716 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:36.204411 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762b6442-d884-4ae5-be83-e3fc394dd716 admin admin] PATH_INFO: `/v3/users/6ec52681fc41477e92786ad61aa96a76` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:36.205996 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762b6442-d884-4ae5-be83-e3fc394dd716 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6ec52681fc41477e92786ad61aa96a76)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:36.215733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762b6442-d884-4ae5-be83-e3fc394dd716 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:36.355387 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-762b6442-d884-4ae5-be83-e3fc394dd716 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6ec52681fc41477e92786ad61aa96a76'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:36.370635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-762b6442-d884-4ae5-be83-e3fc394dd716 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6ec52681fc41477e92786ad61aa96a76'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:36.397626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-762b6442-d884-4ae5-be83-e3fc394dd716 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6ec52681fc41477e92786ad61aa96a76'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:36.424183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-762b6442-d884-4ae5-be83-e3fc394dd716 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6ec52681fc41477e92786ad61aa96a76'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:36.425130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-762b6442-d884-4ae5-be83-e3fc394dd716 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6ec52681fc41477e92786ad61aa96a76'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:36.442928 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1187/2375] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:36:36 2022] DELETE /identity/v3/users/6ec52681fc41477e92786ad61aa96a76 => generated 0 bytes in 414 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:36.763754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bef174-99e0-484d-abd8-86a94bedc477 service swift] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:36.788812 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bef174-99e0-484d-abd8-86a94bedc477 service swift] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:36.900138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b652d50-f3a6-4cf3-b410-5b6a430b9872 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:36.902057 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b652d50-f3a6-4cf3-b410-5b6a430b9872 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:36.902350 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b652d50-f3a6-4cf3-b410-5b6a430b9872 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:36.902646 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b652d50-f3a6-4cf3-b410-5b6a430b9872 admin admin] PATH_INFO: `/v3/projects/742e9ad0995f4f96bcc1f75f5689a590` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:36.903880 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b652d50-f3a6-4cf3-b410-5b6a430b9872 admin admin] RBAC: Authorizing `identity:delete_project(project_id=742e9ad0995f4f96bcc1f75f5689a590)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:36.906737 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b652d50-f3a6-4cf3-b410-5b6a430b9872 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:37.045244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3b652d50-f3a6-4cf3-b410-5b6a430b9872 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '742e9ad0995f4f96bcc1f75f5689a590', 'request_id': 'req-3b652d50-f3a6-4cf3-b410-5b6a430b9872'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:37.055284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3b652d50-f3a6-4cf3-b410-5b6a430b9872 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '742e9ad0995f4f96bcc1f75f5689a590', 'request_id': 'req-3b652d50-f3a6-4cf3-b410-5b6a430b9872'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:37.070363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3b652d50-f3a6-4cf3-b410-5b6a430b9872 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '742e9ad0995f4f96bcc1f75f5689a590', 'request_id': 'req-3b652d50-f3a6-4cf3-b410-5b6a430b9872'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:37.105303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3b652d50-f3a6-4cf3-b410-5b6a430b9872 admin admin] The token cache is being invalidate because project 742e9ad0995f4f96bcc1f75f5689a590 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:36:37.106003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3b652d50-f3a6-4cf3-b410-5b6a430b9872 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:37.107119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-3b652d50-f3a6-4cf3-b410-5b6a430b9872 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:37.120439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1189/2376] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:36:36 2022] DELETE /identity/v3/projects/742e9ad0995f4f96bcc1f75f5689a590 => generated 0 bytes in 347 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:39.656870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762b6442-d884-4ae5-be83-e3fc394dd716 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:39.658739 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-762b6442-d884-4ae5-be83-e3fc394dd716 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:39.684822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762b6442-d884-4ae5-be83-e3fc394dd716 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:39.745888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:39.774631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9cb52e-e4fa-4fa5-8d7f-f2232af21772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:39.775454 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9cb52e-e4fa-4fa5-8d7f-f2232af21772 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:39.776199 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9cb52e-e4fa-4fa5-8d7f-f2232af21772 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:39.776685 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9cb52e-e4fa-4fa5-8d7f-f2232af21772 admin admin] PATH_INFO: `/v3/users/c4e8ecc72ee24f3b88339a5fee7f3ccc` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:39.778876 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9cb52e-e4fa-4fa5-8d7f-f2232af21772 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c4e8ecc72ee24f3b88339a5fee7f3ccc)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:39.781468 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9cb52e-e4fa-4fa5-8d7f-f2232af21772 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:39.859718 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-1e9cb52e-e4fa-4fa5-8d7f-f2232af21772 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c4e8ecc72ee24f3b88339a5fee7f3ccc'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:39.873909 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-1e9cb52e-e4fa-4fa5-8d7f-f2232af21772 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c4e8ecc72ee24f3b88339a5fee7f3ccc'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:39.884465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-1e9cb52e-e4fa-4fa5-8d7f-f2232af21772 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c4e8ecc72ee24f3b88339a5fee7f3ccc'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:39.909202 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-1e9cb52e-e4fa-4fa5-8d7f-f2232af21772 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c4e8ecc72ee24f3b88339a5fee7f3ccc'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:39.910042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-1e9cb52e-e4fa-4fa5-8d7f-f2232af21772 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c4e8ecc72ee24f3b88339a5fee7f3ccc'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:39.923146 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1188/2377] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:36:39 2022] DELETE /identity/v3/users/c4e8ecc72ee24f3b88339a5fee7f3ccc => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:39.928864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1169bd-e33a-4731-9f45-b506edec580f None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:39.929342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1169bd-e33a-4731-9f45-b506edec580f None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:39.929442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1169bd-e33a-4731-9f45-b506edec580f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:39.958625 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-4e1169bd-e33a-4731-9f45-b506edec580f None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:39.960395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-4e1169bd-e33a-4731-9f45-b506edec580f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:40.091320 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1190/2378] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:40.106990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9cb52e-e4fa-4fa5-8d7f-f2232af21772 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:40.109329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1e9cb52e-e4fa-4fa5-8d7f-f2232af21772 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:40.215719 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9cb52e-e4fa-4fa5-8d7f-f2232af21772 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:40.222559 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36b99f0-a6c7-4d26-95fc-147bb90d6d3c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:40.224122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36b99f0-a6c7-4d26-95fc-147bb90d6d3c service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:40.224548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36b99f0-a6c7-4d26-95fc-147bb90d6d3c service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:40.225437 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36b99f0-a6c7-4d26-95fc-147bb90d6d3c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:40.239546 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36b99f0-a6c7-4d26-95fc-147bb90d6d3c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:40.241364 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36b99f0-a6c7-4d26-95fc-147bb90d6d3c service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:40.256775 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1189/2379] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:36:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:36:40.718176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1169bd-e33a-4731-9f45-b506edec580f None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:40.720232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-4e1169bd-e33a-4731-9f45-b506edec580f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:40.788369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1169bd-e33a-4731-9f45-b506edec580f None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:40.796716 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d375bf-909c-48cb-a1fc-232ede9ae183 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:40.797523 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d375bf-909c-48cb-a1fc-232ede9ae183 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:40.797905 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d375bf-909c-48cb-a1fc-232ede9ae183 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:40.798287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d375bf-909c-48cb-a1fc-232ede9ae183 admin admin] PATH_INFO: `/v3/projects/d376e1791911493aac6eb714bd2b3bcf` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:40.800425 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d375bf-909c-48cb-a1fc-232ede9ae183 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d376e1791911493aac6eb714bd2b3bcf)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:40.802464 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d375bf-909c-48cb-a1fc-232ede9ae183 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:40.832414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:40.883732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-17d375bf-909c-48cb-a1fc-232ede9ae183 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd376e1791911493aac6eb714bd2b3bcf', 'request_id': 'req-17d375bf-909c-48cb-a1fc-232ede9ae183'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:40.894420 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-17d375bf-909c-48cb-a1fc-232ede9ae183 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd376e1791911493aac6eb714bd2b3bcf', 'request_id': 'req-17d375bf-909c-48cb-a1fc-232ede9ae183'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:40.903682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-17d375bf-909c-48cb-a1fc-232ede9ae183 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd376e1791911493aac6eb714bd2b3bcf', 'request_id': 'req-17d375bf-909c-48cb-a1fc-232ede9ae183'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:40.927107 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-17d375bf-909c-48cb-a1fc-232ede9ae183 admin admin] The token cache is being invalidate because project d376e1791911493aac6eb714bd2b3bcf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:36:40.927815 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-17d375bf-909c-48cb-a1fc-232ede9ae183 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:40.929029 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-17d375bf-909c-48cb-a1fc-232ede9ae183 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:40.932887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1191/2380] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:36:40 2022] DELETE /identity/v3/projects/d376e1791911493aac6eb714bd2b3bcf => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:40.937855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feebca77-393d-4c39-b7aa-da2ffb25bee7 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:40.938358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feebca77-393d-4c39-b7aa-da2ffb25bee7 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:40.938516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feebca77-393d-4c39-b7aa-da2ffb25bee7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:40.963623 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-feebca77-393d-4c39-b7aa-da2ffb25bee7 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:40.965239 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-feebca77-393d-4c39-b7aa-da2ffb25bee7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:41.050117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1190/2381] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:36:41.059695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f75612-e73e-46ec-a912-56f85472f7f4 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:41.060174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f75612-e73e-46ec-a912-56f85472f7f4 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:41.060711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f75612-e73e-46ec-a912-56f85472f7f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:41.099965 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-60f75612-e73e-46ec-a912-56f85472f7f4 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:41.107385 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-60f75612-e73e-46ec-a912-56f85472f7f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:41.125873 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1192/2382] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:41.131395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feebca77-393d-4c39-b7aa-da2ffb25bee7 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:41.139374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:41.153097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feebca77-393d-4c39-b7aa-da2ffb25bee7 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:41.161909 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a8aa19-6ec7-411e-b6f7-5906231dfffc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:41.162949 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a8aa19-6ec7-411e-b6f7-5906231dfffc admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:41.163476 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a8aa19-6ec7-411e-b6f7-5906231dfffc admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:41.164085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a8aa19-6ec7-411e-b6f7-5906231dfffc admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:41.165235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a8aa19-6ec7-411e-b6f7-5906231dfffc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:36:41.165667 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a8aa19-6ec7-411e-b6f7-5906231dfffc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:41.169885 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a8aa19-6ec7-411e-b6f7-5906231dfffc admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:41.188388 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1191/2383] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:36:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:36:41.192835 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f75612-e73e-46ec-a912-56f85472f7f4 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:41.207817 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f75612-e73e-46ec-a912-56f85472f7f4 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:41.218535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e3c0f8-3c0c-4cd8-86d8-51b4cd3bb080 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:41.219505 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e3c0f8-3c0c-4cd8-86d8-51b4cd3bb080 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:41.219979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e3c0f8-3c0c-4cd8-86d8-51b4cd3bb080 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:41.220520 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e3c0f8-3c0c-4cd8-86d8-51b4cd3bb080 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:41.221638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e3c0f8-3c0c-4cd8-86d8-51b4cd3bb080 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:41.223735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e3c0f8-3c0c-4cd8-86d8-51b4cd3bb080 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:41.270104 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1193/2384] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:36:41 2022] POST /identity/v3/projects => generated 375 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:36:41.274438 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a8aa19-6ec7-411e-b6f7-5906231dfffc admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:41.287124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a8aa19-6ec7-411e-b6f7-5906231dfffc admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:41.412648 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18eb8d8f-b38e-4113-b789-84db5d63e2bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:41.413272 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18eb8d8f-b38e-4113-b789-84db5d63e2bf admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:41.413550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18eb8d8f-b38e-4113-b789-84db5d63e2bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:41.413824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18eb8d8f-b38e-4113-b789-84db5d63e2bf admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:41.414466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18eb8d8f-b38e-4113-b789-84db5d63e2bf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:41.417342 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18eb8d8f-b38e-4113-b789-84db5d63e2bf admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:41.478341 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1192/2385] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:36:41 2022] POST /identity/v3/users => generated 288 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:36:41.483031 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e3c0f8-3c0c-4cd8-86d8-51b4cd3bb080 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:41.500007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e3c0f8-3c0c-4cd8-86d8-51b4cd3bb080 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:41.505453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffaf6a3-fc98-4865-bfaa-bf1923e8aa5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:41.506081 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffaf6a3-fc98-4865-bfaa-bf1923e8aa5b admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:41.506367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffaf6a3-fc98-4865-bfaa-bf1923e8aa5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:41.506737 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffaf6a3-fc98-4865-bfaa-bf1923e8aa5b admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:41.507430 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffaf6a3-fc98-4865-bfaa-bf1923e8aa5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:41.509086 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffaf6a3-fc98-4865-bfaa-bf1923e8aa5b admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:41.522932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1194/2386] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:36:41 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:36:41.528953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18eb8d8f-b38e-4113-b789-84db5d63e2bf admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:41.543992 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18eb8d8f-b38e-4113-b789-84db5d63e2bf admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:41.550918 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03a2a9d-f7f2-4122-a073-d0384d22bfc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:41.551745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03a2a9d-f7f2-4122-a073-d0384d22bfc9 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:41.552122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03a2a9d-f7f2-4122-a073-d0384d22bfc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:41.552539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03a2a9d-f7f2-4122-a073-d0384d22bfc9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:41.553312 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03a2a9d-f7f2-4122-a073-d0384d22bfc9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:41.555249 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03a2a9d-f7f2-4122-a073-d0384d22bfc9 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:41.575235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1193/2387] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:36:41 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:36:41.583032 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffaf6a3-fc98-4865-bfaa-bf1923e8aa5b admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:41.602672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffaf6a3-fc98-4865-bfaa-bf1923e8aa5b admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:41.615499 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265d8ffe-5711-4424-a1ff-a12d88c76744 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:41.616103 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265d8ffe-5711-4424-a1ff-a12d88c76744 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:41.616444 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265d8ffe-5711-4424-a1ff-a12d88c76744 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:41.616775 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265d8ffe-5711-4424-a1ff-a12d88c76744 admin admin] PATH_INFO: `/v3/projects/75243fd6a7fc48389dbeaf55b6d312bd/users/d827fdfdbc864eb8ad6019adca86ef8a/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:41.645067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265d8ffe-5711-4424-a1ff-a12d88c76744 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75243fd6a7fc48389dbeaf55b6d312bd, user_id=d827fdfdbc864eb8ad6019adca86ef8a, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:41.647946 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265d8ffe-5711-4424-a1ff-a12d88c76744 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:41.682897 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1195/2388] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:36:41 2022] PUT /identity/v3/projects/75243fd6a7fc48389dbeaf55b6d312bd/users/d827fdfdbc864eb8ad6019adca86ef8a/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:41.697401 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d43f956-8c28-4b04-94c4-4d402e608bb3 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:41.698650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d43f956-8c28-4b04-94c4-4d402e608bb3 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:41.699097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d43f956-8c28-4b04-94c4-4d402e608bb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:41.746289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-8d43f956-8c28-4b04-94c4-4d402e608bb3 None None] MFA Rules not processed for user `d827fdfdbc864eb8ad6019adca86ef8a`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:41.747768 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-8d43f956-8c28-4b04-94c4-4d402e608bb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:41.924629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1194/2389] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:41 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:36:41.933746 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265d8ffe-5711-4424-a1ff-a12d88c76744 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:41.939462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-265d8ffe-5711-4424-a1ff-a12d88c76744 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:42.039898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265d8ffe-5711-4424-a1ff-a12d88c76744 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:42.045388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da30811b-9e3c-42df-98a5-746de16b4c9a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:42.046162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da30811b-9e3c-42df-98a5-746de16b4c9a service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:42.046452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da30811b-9e3c-42df-98a5-746de16b4c9a service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:42.046726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da30811b-9e3c-42df-98a5-746de16b4c9a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:42.059731 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da30811b-9e3c-42df-98a5-746de16b4c9a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:42.061453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da30811b-9e3c-42df-98a5-746de16b4c9a service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:42.073895 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1196/2390] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:36:41 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:36:45.548190 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d43f956-8c28-4b04-94c4-4d402e608bb3 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:45.618433 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d43f956-8c28-4b04-94c4-4d402e608bb3 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:45.623773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c6cdca-1b5e-4048-80cf-18f96c65af6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:45.624487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c6cdca-1b5e-4048-80cf-18f96c65af6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:45.624767 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c6cdca-1b5e-4048-80cf-18f96c65af6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:45.625037 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c6cdca-1b5e-4048-80cf-18f96c65af6a admin admin] PATH_INFO: `/v3/users/d827fdfdbc864eb8ad6019adca86ef8a` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:45.626188 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c6cdca-1b5e-4048-80cf-18f96c65af6a admin admin] RBAC: Authorizing `identity:delete_user(user_id=d827fdfdbc864eb8ad6019adca86ef8a)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:45.628126 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c6cdca-1b5e-4048-80cf-18f96c65af6a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:45.630430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:45.687999 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c1c6cdca-1b5e-4048-80cf-18f96c65af6a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd827fdfdbc864eb8ad6019adca86ef8a'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:45.706192 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c1c6cdca-1b5e-4048-80cf-18f96c65af6a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd827fdfdbc864eb8ad6019adca86ef8a'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:45.715343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c1c6cdca-1b5e-4048-80cf-18f96c65af6a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd827fdfdbc864eb8ad6019adca86ef8a'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:45.731506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c1c6cdca-1b5e-4048-80cf-18f96c65af6a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd827fdfdbc864eb8ad6019adca86ef8a'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:45.732642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c1c6cdca-1b5e-4048-80cf-18f96c65af6a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd827fdfdbc864eb8ad6019adca86ef8a'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:45.744948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1195/2391] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:36:45 2022] DELETE /identity/v3/users/d827fdfdbc864eb8ad6019adca86ef8a => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:45.750948 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488badb9-f7db-4b06-b84a-bf0a74d08513 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:45.751781 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488badb9-f7db-4b06-b84a-bf0a74d08513 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:45.751781 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488badb9-f7db-4b06-b84a-bf0a74d08513 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:45.771390 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-488badb9-f7db-4b06-b84a-bf0a74d08513 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:45.775082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-488badb9-f7db-4b06-b84a-bf0a74d08513 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:45.824681 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:45.858097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1197/2392] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:45.870469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c6cdca-1b5e-4048-80cf-18f96c65af6a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:45.872018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c1c6cdca-1b5e-4048-80cf-18f96c65af6a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:45.942266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c6cdca-1b5e-4048-80cf-18f96c65af6a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:45.946624 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81233df3-7d58-4813-bf96-46b06a729bd4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:45.947246 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81233df3-7d58-4813-bf96-46b06a729bd4 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:45.947873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81233df3-7d58-4813-bf96-46b06a729bd4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:45.948059 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81233df3-7d58-4813-bf96-46b06a729bd4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:45.959425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81233df3-7d58-4813-bf96-46b06a729bd4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:45.960791 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81233df3-7d58-4813-bf96-46b06a729bd4 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:45.974648 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1196/2393] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:36:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:36:46.271831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488badb9-f7db-4b06-b84a-bf0a74d08513 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:46.273682 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-488badb9-f7db-4b06-b84a-bf0a74d08513 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:46.301607 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488badb9-f7db-4b06-b84a-bf0a74d08513 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:46.307777 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603a60d0-557f-4869-af43-0606c62a3f99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:46.308630 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603a60d0-557f-4869-af43-0606c62a3f99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:46.309023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603a60d0-557f-4869-af43-0606c62a3f99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:46.309352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603a60d0-557f-4869-af43-0606c62a3f99 admin admin] PATH_INFO: `/v3/projects/75243fd6a7fc48389dbeaf55b6d312bd` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:46.310676 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603a60d0-557f-4869-af43-0606c62a3f99 admin admin] RBAC: Authorizing `identity:delete_project(project_id=75243fd6a7fc48389dbeaf55b6d312bd)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:46.312801 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603a60d0-557f-4869-af43-0606c62a3f99 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:46.369993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-603a60d0-557f-4869-af43-0606c62a3f99 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75243fd6a7fc48389dbeaf55b6d312bd', 'request_id': 'req-603a60d0-557f-4869-af43-0606c62a3f99'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:46.376786 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-603a60d0-557f-4869-af43-0606c62a3f99 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75243fd6a7fc48389dbeaf55b6d312bd', 'request_id': 'req-603a60d0-557f-4869-af43-0606c62a3f99'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:46.386184 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-603a60d0-557f-4869-af43-0606c62a3f99 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75243fd6a7fc48389dbeaf55b6d312bd', 'request_id': 'req-603a60d0-557f-4869-af43-0606c62a3f99'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:46.403616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-603a60d0-557f-4869-af43-0606c62a3f99 admin admin] The token cache is being invalidate because project 75243fd6a7fc48389dbeaf55b6d312bd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:36:46.403910 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-603a60d0-557f-4869-af43-0606c62a3f99 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:46.404589 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-603a60d0-557f-4869-af43-0606c62a3f99 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:36:46.406364 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1198/2394] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:36:46 2022] DELETE /identity/v3/projects/75243fd6a7fc48389dbeaf55b6d312bd => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:36:46.411503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6419bff4-feac-4933-b869-e56f1c800c7e None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:46.411794 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6419bff4-feac-4933-b869-e56f1c800c7e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:46.412069 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6419bff4-feac-4933-b869-e56f1c800c7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:46.433766 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-6419bff4-feac-4933-b869-e56f1c800c7e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:46.434988 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6419bff4-feac-4933-b869-e56f1c800c7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:46.499999 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1197/2395] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:46.508162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97626424-fad0-4e11-b2c8-c5bb8a36785e None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:46.508493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97626424-fad0-4e11-b2c8-c5bb8a36785e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:46.508772 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97626424-fad0-4e11-b2c8-c5bb8a36785e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:46.530950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-97626424-fad0-4e11-b2c8-c5bb8a36785e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:46.532183 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-97626424-fad0-4e11-b2c8-c5bb8a36785e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:46.540128 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1199/2396] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:46.545014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6419bff4-feac-4933-b869-e56f1c800c7e None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:46.555019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6419bff4-feac-4933-b869-e56f1c800c7e None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:46.560434 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b7d947-f017-44d6-9f53-f04b361d8399 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:46.561039 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b7d947-f017-44d6-9f53-f04b361d8399 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:46.561310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b7d947-f017-44d6-9f53-f04b361d8399 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:46.561605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b7d947-f017-44d6-9f53-f04b361d8399 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:46.562242 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b7d947-f017-44d6-9f53-f04b361d8399 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:36:46.562541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b7d947-f017-44d6-9f53-f04b361d8399 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:46.564120 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b7d947-f017-44d6-9f53-f04b361d8399 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:46.576795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1198/2397] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:36:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:36:46.581553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97626424-fad0-4e11-b2c8-c5bb8a36785e None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:46.592058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97626424-fad0-4e11-b2c8-c5bb8a36785e None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:46.596936 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265cabdd-9241-45e7-b21e-0bf48d7f7da7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:46.597612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265cabdd-9241-45e7-b21e-0bf48d7f7da7 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:46.598005 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265cabdd-9241-45e7-b21e-0bf48d7f7da7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:46.598387 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265cabdd-9241-45e7-b21e-0bf48d7f7da7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:46.599138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265cabdd-9241-45e7-b21e-0bf48d7f7da7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:46.601207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265cabdd-9241-45e7-b21e-0bf48d7f7da7 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:46.629611 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1200/2398] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:36:46 2022] POST /identity/v3/projects => generated 369 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:36:46.634072 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b7d947-f017-44d6-9f53-f04b361d8399 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:46.645974 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b7d947-f017-44d6-9f53-f04b361d8399 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:46.652222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:46.731162 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2d01cc-2a96-42c0-a306-9381a8955e01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:46.731947 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2d01cc-2a96-42c0-a306-9381a8955e01 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:46.732266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2d01cc-2a96-42c0-a306-9381a8955e01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:46.732641 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2d01cc-2a96-42c0-a306-9381a8955e01 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:46.733359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec2d01cc-2a96-42c0-a306-9381a8955e01 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:46.735440 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec2d01cc-2a96-42c0-a306-9381a8955e01 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:46.769451 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1199/2399] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:36:46 2022] POST /identity/v3/users => generated 285 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:36:46.771401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265cabdd-9241-45e7-b21e-0bf48d7f7da7 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:46.790006 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265cabdd-9241-45e7-b21e-0bf48d7f7da7 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:46.796046 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610f4040-171c-4b40-9827-70ca2834f6f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:46.796731 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610f4040-171c-4b40-9827-70ca2834f6f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:46.796978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610f4040-171c-4b40-9827-70ca2834f6f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:46.797284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610f4040-171c-4b40-9827-70ca2834f6f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:46.797936 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610f4040-171c-4b40-9827-70ca2834f6f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:46.799551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610f4040-171c-4b40-9827-70ca2834f6f4 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:46.812333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1201/2400] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:36:46 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:36:46.816817 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2d01cc-2a96-42c0-a306-9381a8955e01 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:46.828398 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2d01cc-2a96-42c0-a306-9381a8955e01 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:46.839604 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4069bde5-c3e0-4f46-9d33-8b8a9ca28204 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:46.841662 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4069bde5-c3e0-4f46-9d33-8b8a9ca28204 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:46.842173 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4069bde5-c3e0-4f46-9d33-8b8a9ca28204 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:46.842678 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4069bde5-c3e0-4f46-9d33-8b8a9ca28204 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:46.844642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4069bde5-c3e0-4f46-9d33-8b8a9ca28204 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:46.847470 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4069bde5-c3e0-4f46-9d33-8b8a9ca28204 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:46.876381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1200/2401] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:36:46 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:36:46.878414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610f4040-171c-4b40-9827-70ca2834f6f4 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:46.896647 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610f4040-171c-4b40-9827-70ca2834f6f4 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:46.903924 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7818a630-040d-4f81-ba9c-d28d47256e34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:46.905389 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7818a630-040d-4f81-ba9c-d28d47256e34 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:46.905857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7818a630-040d-4f81-ba9c-d28d47256e34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:46.906262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7818a630-040d-4f81-ba9c-d28d47256e34 admin admin] PATH_INFO: `/v3/projects/47c2da5b0ed14916bbed7796f8575d10/users/53a0c6b231364be59b9c57493d3d821d/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:46.927961 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7818a630-040d-4f81-ba9c-d28d47256e34 admin admin] RBAC: Authorizing `identity:create_grant(project_id=47c2da5b0ed14916bbed7796f8575d10, user_id=53a0c6b231364be59b9c57493d3d821d, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:46.930821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7818a630-040d-4f81-ba9c-d28d47256e34 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:46.946129 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1202/2402] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:36:46 2022] PUT /identity/v3/projects/47c2da5b0ed14916bbed7796f8575d10/users/53a0c6b231364be59b9c57493d3d821d/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:36:46.958183 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c64ca3-a030-447a-9ecf-736a66844a85 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:46.958653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c64ca3-a030-447a-9ecf-736a66844a85 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:46.958982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c64ca3-a030-447a-9ecf-736a66844a85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:46.991533 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-82c64ca3-a030-447a-9ecf-736a66844a85 None None] MFA Rules not processed for user `53a0c6b231364be59b9c57493d3d821d`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:46.993154 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-82c64ca3-a030-447a-9ecf-736a66844a85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:47.077624 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1201/2403] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:36:46 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:47.092678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7818a630-040d-4f81-ba9c-d28d47256e34 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:47.094439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7818a630-040d-4f81-ba9c-d28d47256e34 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:47.157436 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7818a630-040d-4f81-ba9c-d28d47256e34 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:47.161281 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816e1ab4-70c5-4f00-a4ed-331b7f7a4c2f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35a5a4ef40204e158649ca25583bbde7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:47.161903 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816e1ab4-70c5-4f00-a4ed-331b7f7a4c2f service glance] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:47.162188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816e1ab4-70c5-4f00-a4ed-331b7f7a4c2f service glance] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:47.162486 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816e1ab4-70c5-4f00-a4ed-331b7f7a4c2f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:47.172282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816e1ab4-70c5-4f00-a4ed-331b7f7a4c2f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:47.173756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816e1ab4-70c5-4f00-a4ed-331b7f7a4c2f service glance] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:47.184186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1203/2404] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:36:47 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:36:53.569842 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6580f90c-46a5-4d16-a46d-86b203615618 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:53.570546 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6580f90c-46a5-4d16-a46d-86b203615618 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:53.571153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6580f90c-46a5-4d16-a46d-86b203615618 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:53.608576 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-6580f90c-46a5-4d16-a46d-86b203615618 None None] MFA Rules not processed for user `34afa4aac10c4a018ca05a0ba34c940a`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:36:53.616132 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6580f90c-46a5-4d16-a46d-86b203615618 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:53.769666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1202/2405] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Oct 4 14:36:53 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:36:53.773557 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816e1ab4-70c5-4f00-a4ed-331b7f7a4c2f service glance] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:36:53.776245 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-816e1ab4-70c5-4f00-a4ed-331b7f7a4c2f service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:36:53.785396 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:36:53.995416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816e1ab4-70c5-4f00-a4ed-331b7f7a4c2f service glance] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:36:54.006225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed117019-57b7-4307-9889-c471216498b3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5980e353164c4873bcdd2af847e0a1f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:36:54.006886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed117019-57b7-4307-9889-c471216498b3 service swift] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:36:54.007170 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed117019-57b7-4307-9889-c471216498b3 service swift] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:36:54.007437 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed117019-57b7-4307-9889-c471216498b3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:36:54.018562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed117019-57b7-4307-9889-c471216498b3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:36:54.020352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed117019-57b7-4307-9889-c471216498b3 service swift] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:36:54.030750 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1204/2406] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Oct 4 14:36:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 258 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 14:37:03.733246 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=87,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:04.025892 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=30 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:12.030655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6580f90c-46a5-4d16-a46d-86b203615618 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:12.032257 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6580f90c-46a5-4d16-a46d-86b203615618 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:12.115749 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6580f90c-46a5-4d16-a46d-86b203615618 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:12.120723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2287c7d0-83ee-4f1a-bd10-81c5ad021bce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:12.121362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2287c7d0-83ee-4f1a-bd10-81c5ad021bce service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:12.121639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2287c7d0-83ee-4f1a-bd10-81c5ad021bce service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:12.121905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2287c7d0-83ee-4f1a-bd10-81c5ad021bce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:12.127569 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-2287c7d0-83ee-4f1a-bd10-81c5ad021bce service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:12.245454 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2287c7d0-83ee-4f1a-bd10-81c5ad021bce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:12.246962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2287c7d0-83ee-4f1a-bd10-81c5ad021bce service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:12.260475 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1203/2407] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 231 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:37:13.510893 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88a2f3e-c5fe-4a34-b52d-ae3ce59a8294 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:13.511377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88a2f3e-c5fe-4a34-b52d-ae3ce59a8294 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:13.511499 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88a2f3e-c5fe-4a34-b52d-ae3ce59a8294 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:13.531775 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-c88a2f3e-c5fe-4a34-b52d-ae3ce59a8294 None None] MFA Rules not processed for user `34afa4aac10c4a018ca05a0ba34c940a`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:13.533029 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c88a2f3e-c5fe-4a34-b52d-ae3ce59a8294 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:13.540855 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1205/2408] 10.222.0.64 () {58 vars in 967 bytes} [Tue Oct 4 14:37:13 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:13.554324 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2287c7d0-83ee-4f1a-bd10-81c5ad021bce service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:13.578572 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2287c7d0-83ee-4f1a-bd10-81c5ad021bce service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:13.584773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4ac980-23dc-453b-ae01-e851972a0f28 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5980e353164c4873bcdd2af847e0a1f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:13.585660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4ac980-23dc-453b-ae01-e851972a0f28 service swift] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:13.586139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4ac980-23dc-453b-ae01-e851972a0f28 service swift] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:13.586616 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4ac980-23dc-453b-ae01-e851972a0f28 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:13.599232 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4ac980-23dc-453b-ae01-e851972a0f28 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:13.600918 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4ac980-23dc-453b-ae01-e851972a0f28 service swift] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:13.612002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1204/2409] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Oct 4 14:37:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 14:37:15.058878 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07be7667-1308-4c3d-b33d-caf335d2590f None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:15.059883 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07be7667-1308-4c3d-b33d-caf335d2590f None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:15.060168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07be7667-1308-4c3d-b33d-caf335d2590f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:15.084632 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-07be7667-1308-4c3d-b33d-caf335d2590f None None] MFA Rules not processed for user `34afa4aac10c4a018ca05a0ba34c940a`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:15.086137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-07be7667-1308-4c3d-b33d-caf335d2590f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:15.110135 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1206/2410] 10.222.0.64 () {58 vars in 967 bytes} [Tue Oct 4 14:37:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:15.111173 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4ac980-23dc-453b-ae01-e851972a0f28 service swift] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:15.130901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4ac980-23dc-453b-ae01-e851972a0f28 service swift] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:15.136823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cd54dc-6b2f-4668-9173-627305a4edf0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5980e353164c4873bcdd2af847e0a1f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:15.137439 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cd54dc-6b2f-4668-9173-627305a4edf0 service swift] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:15.137711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cd54dc-6b2f-4668-9173-627305a4edf0 service swift] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:15.137975 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cd54dc-6b2f-4668-9173-627305a4edf0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:15.147774 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cd54dc-6b2f-4668-9173-627305a4edf0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:15.149616 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cd54dc-6b2f-4668-9173-627305a4edf0 service swift] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:15.164503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1205/2411] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Oct 4 14:37:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 14:37:16.517318 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07be7667-1308-4c3d-b33d-caf335d2590f None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:16.519567 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-07be7667-1308-4c3d-b33d-caf335d2590f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:16.578464 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cd54dc-6b2f-4668-9173-627305a4edf0 service swift] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:16.580654 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-40cd54dc-6b2f-4668-9173-627305a4edf0 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:16.603045 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cd54dc-6b2f-4668-9173-627305a4edf0 service swift] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:16.609072 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686a5648-9065-4af0-8895-3e15247ad540 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:16.609766 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686a5648-9065-4af0-8895-3e15247ad540 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:16.610213 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686a5648-9065-4af0-8895-3e15247ad540 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:16.610414 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686a5648-9065-4af0-8895-3e15247ad540 admin admin] PATH_INFO: `/v3/users/fc02bb4b8a32400faef8743d2418a648` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:16.612564 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686a5648-9065-4af0-8895-3e15247ad540 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc02bb4b8a32400faef8743d2418a648)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:16.614471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686a5648-9065-4af0-8895-3e15247ad540 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:16.634103 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07be7667-1308-4c3d-b33d-caf335d2590f None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:16.641011 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28aa36db-9d65-408b-842b-0fd837e957ef service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '436deb3d60c54a799700986519b2ef68', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:16.641645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28aa36db-9d65-408b-842b-0fd837e957ef service placement] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:16.641982 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28aa36db-9d65-408b-842b-0fd837e957ef service placement] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:16.642168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28aa36db-9d65-408b-842b-0fd837e957ef service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:16.644583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-28aa36db-9d65-408b-842b-0fd837e957ef service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:16.662811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28aa36db-9d65-408b-842b-0fd837e957ef service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:16.664634 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28aa36db-9d65-408b-842b-0fd837e957ef service placement] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:16.676639 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1207/2412] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:16 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:16.681354 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-686a5648-9065-4af0-8895-3e15247ad540 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc02bb4b8a32400faef8743d2418a648'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:16.693017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-686a5648-9065-4af0-8895-3e15247ad540 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc02bb4b8a32400faef8743d2418a648'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:16.725123 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-686a5648-9065-4af0-8895-3e15247ad540 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc02bb4b8a32400faef8743d2418a648'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:16.737445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:16.738918 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-686a5648-9065-4af0-8895-3e15247ad540 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc02bb4b8a32400faef8743d2418a648'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:16.739757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-686a5648-9065-4af0-8895-3e15247ad540 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc02bb4b8a32400faef8743d2418a648'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:16.749356 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1206/2413] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:16 2022] DELETE /identity/v3/users/fc02bb4b8a32400faef8743d2418a648 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:16.959237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28aa36db-9d65-408b-842b-0fd837e957ef service placement] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:16.975677 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28aa36db-9d65-408b-842b-0fd837e957ef service placement] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:17.057680 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a03c02-6d13-4177-8c1d-5d45edd6de3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:17.059026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a03c02-6d13-4177-8c1d-5d45edd6de3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:17.059310 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a03c02-6d13-4177-8c1d-5d45edd6de3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:17.059746 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a03c02-6d13-4177-8c1d-5d45edd6de3e admin admin] PATH_INFO: `/v3/projects/3a4fa5d0aa0e4bf98816e4600023ca99` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:17.061065 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a03c02-6d13-4177-8c1d-5d45edd6de3e admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a4fa5d0aa0e4bf98816e4600023ca99)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:17.063023 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a03c02-6d13-4177-8c1d-5d45edd6de3e admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:17.130782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a6a03c02-6d13-4177-8c1d-5d45edd6de3e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a4fa5d0aa0e4bf98816e4600023ca99', 'request_id': 'req-a6a03c02-6d13-4177-8c1d-5d45edd6de3e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:17.139452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a6a03c02-6d13-4177-8c1d-5d45edd6de3e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a4fa5d0aa0e4bf98816e4600023ca99', 'request_id': 'req-a6a03c02-6d13-4177-8c1d-5d45edd6de3e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:17.149495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a6a03c02-6d13-4177-8c1d-5d45edd6de3e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a4fa5d0aa0e4bf98816e4600023ca99', 'request_id': 'req-a6a03c02-6d13-4177-8c1d-5d45edd6de3e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:17.165259 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a6a03c02-6d13-4177-8c1d-5d45edd6de3e admin admin] The token cache is being invalidate because project 3a4fa5d0aa0e4bf98816e4600023ca99 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:17.165704 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a6a03c02-6d13-4177-8c1d-5d45edd6de3e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:17.166553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a6a03c02-6d13-4177-8c1d-5d45edd6de3e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:17.168052 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1208/2414] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:16 2022] DELETE /identity/v3/projects/3a4fa5d0aa0e4bf98816e4600023ca99 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:17.173117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063f8494-e070-4a57-ad05-a6d883d90162 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:17.173404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063f8494-e070-4a57-ad05-a6d883d90162 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:17.173702 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063f8494-e070-4a57-ad05-a6d883d90162 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:17.194527 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-063f8494-e070-4a57-ad05-a6d883d90162 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:17.196296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-063f8494-e070-4a57-ad05-a6d883d90162 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:17.261273 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1207/2415] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:17.269114 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aad845-6569-4a4f-8e47-3dca8dc24045 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:17.269402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aad845-6569-4a4f-8e47-3dca8dc24045 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:17.269707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aad845-6569-4a4f-8e47-3dca8dc24045 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:17.285073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:17.291960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-39aad845-6569-4a4f-8e47-3dca8dc24045 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:17.293785 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-39aad845-6569-4a4f-8e47-3dca8dc24045 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:17.303364 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1209/2416] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:17.308870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063f8494-e070-4a57-ad05-a6d883d90162 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:17.318871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063f8494-e070-4a57-ad05-a6d883d90162 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:17.323882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9f215e-9597-4fb1-985a-b7d2eb16315a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:17.324512 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9f215e-9597-4fb1-985a-b7d2eb16315a admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:17.324787 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9f215e-9597-4fb1-985a-b7d2eb16315a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:17.325055 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9f215e-9597-4fb1-985a-b7d2eb16315a admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:17.325696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a9f215e-9597-4fb1-985a-b7d2eb16315a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:37:17.325996 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a9f215e-9597-4fb1-985a-b7d2eb16315a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:17.327603 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a9f215e-9597-4fb1-985a-b7d2eb16315a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:17.339819 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1208/2417] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:37:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:37:17.344139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aad845-6569-4a4f-8e47-3dca8dc24045 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:17.354114 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aad845-6569-4a4f-8e47-3dca8dc24045 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:17.358972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a15a8d-d79c-4d66-8dc2-7bb220833229 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:17.359580 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a15a8d-d79c-4d66-8dc2-7bb220833229 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:17.359864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a15a8d-d79c-4d66-8dc2-7bb220833229 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:17.360141 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a15a8d-d79c-4d66-8dc2-7bb220833229 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:17.360811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a15a8d-d79c-4d66-8dc2-7bb220833229 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:17.362726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a15a8d-d79c-4d66-8dc2-7bb220833229 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:17.393939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1210/2418] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:17 2022] POST /identity/v3/projects => generated 375 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:17.398037 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9f215e-9597-4fb1-985a-b7d2eb16315a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:17.408389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9f215e-9597-4fb1-985a-b7d2eb16315a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:17.473910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac74020a-0106-4452-b0d2-c5594462f405 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:17.474629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac74020a-0106-4452-b0d2-c5594462f405 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:17.474905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac74020a-0106-4452-b0d2-c5594462f405 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:17.475271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac74020a-0106-4452-b0d2-c5594462f405 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:17.475807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac74020a-0106-4452-b0d2-c5594462f405 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:17.478370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac74020a-0106-4452-b0d2-c5594462f405 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:17.520607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1209/2419] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:17 2022] POST /identity/v3/users => generated 288 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:37:17.521223 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a15a8d-d79c-4d66-8dc2-7bb220833229 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:17.534549 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a15a8d-d79c-4d66-8dc2-7bb220833229 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:17.541518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae1e128-b4f1-4b4d-a7db-7a66a6477c35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:17.542176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae1e128-b4f1-4b4d-a7db-7a66a6477c35 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:17.542464 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae1e128-b4f1-4b4d-a7db-7a66a6477c35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:17.542744 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae1e128-b4f1-4b4d-a7db-7a66a6477c35 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:17.545585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae1e128-b4f1-4b4d-a7db-7a66a6477c35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:17.548243 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae1e128-b4f1-4b4d-a7db-7a66a6477c35 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:17.570124 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1211/2420] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:17 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:37:17.578058 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac74020a-0106-4452-b0d2-c5594462f405 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:17.590109 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac74020a-0106-4452-b0d2-c5594462f405 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:17.597401 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2b46bf-4743-44c1-8fc8-638eeb50b0ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:17.598099 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2b46bf-4743-44c1-8fc8-638eeb50b0ec admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:17.598480 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2b46bf-4743-44c1-8fc8-638eeb50b0ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:17.598826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2b46bf-4743-44c1-8fc8-638eeb50b0ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:17.599557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2b46bf-4743-44c1-8fc8-638eeb50b0ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:17.601829 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2b46bf-4743-44c1-8fc8-638eeb50b0ec admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:17.618266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1210/2421] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:17 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:17.619984 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae1e128-b4f1-4b4d-a7db-7a66a6477c35 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:17.630574 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae1e128-b4f1-4b4d-a7db-7a66a6477c35 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:17.635761 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f6e253-5df8-4400-afeb-e7e2bb0be29a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:17.636602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f6e253-5df8-4400-afeb-e7e2bb0be29a admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:17.637021 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f6e253-5df8-4400-afeb-e7e2bb0be29a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:17.637396 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f6e253-5df8-4400-afeb-e7e2bb0be29a admin admin] PATH_INFO: `/v3/projects/abcfbd90111d4648b1a2a668bb58912b/users/fdf14ee6f8614addb2a799b45c6a4f6f/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:17.654866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f6e253-5df8-4400-afeb-e7e2bb0be29a admin admin] RBAC: Authorizing `identity:create_grant(project_id=abcfbd90111d4648b1a2a668bb58912b, user_id=fdf14ee6f8614addb2a799b45c6a4f6f, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:17.657748 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f6e253-5df8-4400-afeb-e7e2bb0be29a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:17.674282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1212/2422] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:17 2022] PUT /identity/v3/projects/abcfbd90111d4648b1a2a668bb58912b/users/fdf14ee6f8614addb2a799b45c6a4f6f/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:37:17.686454 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1d00b9-ab5b-4e99-b937-8ae61c29d052 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:17.686738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1d00b9-ab5b-4e99-b937-8ae61c29d052 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:17.687193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1d00b9-ab5b-4e99-b937-8ae61c29d052 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:17.722321 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-cf1d00b9-ab5b-4e99-b937-8ae61c29d052 None None] MFA Rules not processed for user `fdf14ee6f8614addb2a799b45c6a4f6f`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:17.723566 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-cf1d00b9-ab5b-4e99-b937-8ae61c29d052 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:17.799997 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:17.825929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1211/2423] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:17 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:17.839368 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f6e253-5df8-4400-afeb-e7e2bb0be29a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:17.841434 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-14f6e253-5df8-4400-afeb-e7e2bb0be29a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:17.932828 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f6e253-5df8-4400-afeb-e7e2bb0be29a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:17.942263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4061f6-b010-429d-ab63-f6e5aa18a359 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:17.942965 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4061f6-b010-429d-ab63-f6e5aa18a359 service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:17.943307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4061f6-b010-429d-ab63-f6e5aa18a359 service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:17.943651 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4061f6-b010-429d-ab63-f6e5aa18a359 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:17.954874 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4061f6-b010-429d-ab63-f6e5aa18a359 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:17.956733 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4061f6-b010-429d-ab63-f6e5aa18a359 service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:17.969503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1213/2424] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:17 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:18.237630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1d00b9-ab5b-4e99-b937-8ae61c29d052 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:18.315042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1d00b9-ab5b-4e99-b937-8ae61c29d052 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:18.320057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075571d6-4552-466c-b252-b91de8bf78bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:18.320948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075571d6-4552-466c-b252-b91de8bf78bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:18.320948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075571d6-4552-466c-b252-b91de8bf78bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:18.321488 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075571d6-4552-466c-b252-b91de8bf78bb admin admin] PATH_INFO: `/v3/users/fdf14ee6f8614addb2a799b45c6a4f6f` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:18.322749 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075571d6-4552-466c-b252-b91de8bf78bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=fdf14ee6f8614addb2a799b45c6a4f6f)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:18.324633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075571d6-4552-466c-b252-b91de8bf78bb admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:18.378273 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-075571d6-4552-466c-b252-b91de8bf78bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fdf14ee6f8614addb2a799b45c6a4f6f'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:18.388577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-075571d6-4552-466c-b252-b91de8bf78bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fdf14ee6f8614addb2a799b45c6a4f6f'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:18.398265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-075571d6-4552-466c-b252-b91de8bf78bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fdf14ee6f8614addb2a799b45c6a4f6f'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:18.412553 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-075571d6-4552-466c-b252-b91de8bf78bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fdf14ee6f8614addb2a799b45c6a4f6f'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:18.413463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-075571d6-4552-466c-b252-b91de8bf78bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fdf14ee6f8614addb2a799b45c6a4f6f'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:18.420027 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1212/2425] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:18 2022] DELETE /identity/v3/users/fdf14ee6f8614addb2a799b45c6a4f6f => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:18.425111 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3078ae7-22f6-455f-bd4f-07966bed6b16 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:18.425391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3078ae7-22f6-455f-bd4f-07966bed6b16 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:18.425665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3078ae7-22f6-455f-bd4f-07966bed6b16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:18.446735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-c3078ae7-22f6-455f-bd4f-07966bed6b16 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:18.453240 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c3078ae7-22f6-455f-bd4f-07966bed6b16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:18.519585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1214/2426] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:37:18.529796 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075571d6-4552-466c-b252-b91de8bf78bb admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:18.531241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-075571d6-4552-466c-b252-b91de8bf78bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:18.605188 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075571d6-4552-466c-b252-b91de8bf78bb admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:18.610999 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554907ed-401d-4bfe-8fcf-846e0ca101d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:18.611513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554907ed-401d-4bfe-8fcf-846e0ca101d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:18.611767 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554907ed-401d-4bfe-8fcf-846e0ca101d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:18.612087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554907ed-401d-4bfe-8fcf-846e0ca101d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:18.623221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554907ed-401d-4bfe-8fcf-846e0ca101d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:18.624711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554907ed-401d-4bfe-8fcf-846e0ca101d4 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:18.657610 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1213/2427] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:19.003919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3078ae7-22f6-455f-bd4f-07966bed6b16 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:19.006309 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c3078ae7-22f6-455f-bd4f-07966bed6b16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:19.028161 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3078ae7-22f6-455f-bd4f-07966bed6b16 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:19.036014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b396871f-19bb-4056-ac46-255761a2057a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:19.047772 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b396871f-19bb-4056-ac46-255761a2057a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:19.048170 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b396871f-19bb-4056-ac46-255761a2057a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:19.048491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b396871f-19bb-4056-ac46-255761a2057a admin admin] PATH_INFO: `/v3/projects/abcfbd90111d4648b1a2a668bb58912b` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:19.053112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b396871f-19bb-4056-ac46-255761a2057a admin admin] RBAC: Authorizing `identity:delete_project(project_id=abcfbd90111d4648b1a2a668bb58912b)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:19.073377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b396871f-19bb-4056-ac46-255761a2057a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:19.153465 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:19.197352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b396871f-19bb-4056-ac46-255761a2057a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'abcfbd90111d4648b1a2a668bb58912b', 'request_id': 'req-b396871f-19bb-4056-ac46-255761a2057a'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:19.207451 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b396871f-19bb-4056-ac46-255761a2057a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'abcfbd90111d4648b1a2a668bb58912b', 'request_id': 'req-b396871f-19bb-4056-ac46-255761a2057a'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:19.216935 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b396871f-19bb-4056-ac46-255761a2057a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'abcfbd90111d4648b1a2a668bb58912b', 'request_id': 'req-b396871f-19bb-4056-ac46-255761a2057a'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:19.239369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b396871f-19bb-4056-ac46-255761a2057a admin admin] The token cache is being invalidate because project abcfbd90111d4648b1a2a668bb58912b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:19.239668 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b396871f-19bb-4056-ac46-255761a2057a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:19.240927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b396871f-19bb-4056-ac46-255761a2057a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:19.251930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1215/2428] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:19 2022] DELETE /identity/v3/projects/abcfbd90111d4648b1a2a668bb58912b => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:19.257090 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1694aa-8b2d-4ada-a9bc-7d28d183bcd4 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:19.257341 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1694aa-8b2d-4ada-a9bc-7d28d183bcd4 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:19.257808 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1694aa-8b2d-4ada-a9bc-7d28d183bcd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:19.294603 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-7c1694aa-8b2d-4ada-a9bc-7d28d183bcd4 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:19.296231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7c1694aa-8b2d-4ada-a9bc-7d28d183bcd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:19.311329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:19.449017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1214/2429] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:37:19.461317 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8391b315-ec1b-4c1a-bdbf-a5819a6aa44d None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:19.463082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8391b315-ec1b-4c1a-bdbf-a5819a6aa44d None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:19.463645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8391b315-ec1b-4c1a-bdbf-a5819a6aa44d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:19.493850 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-8391b315-ec1b-4c1a-bdbf-a5819a6aa44d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:19.495812 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8391b315-ec1b-4c1a-bdbf-a5819a6aa44d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:19.509978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1216/2430] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:19.514881 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1694aa-8b2d-4ada-a9bc-7d28d183bcd4 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:19.526878 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1694aa-8b2d-4ada-a9bc-7d28d183bcd4 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:19.532637 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba925e52-df13-48f0-8afd-b65788ca63a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:19.533376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba925e52-df13-48f0-8afd-b65788ca63a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:19.533730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba925e52-df13-48f0-8afd-b65788ca63a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:19.534071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba925e52-df13-48f0-8afd-b65788ca63a7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:19.534795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba925e52-df13-48f0-8afd-b65788ca63a7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:37:19.535190 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba925e52-df13-48f0-8afd-b65788ca63a7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:19.537334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba925e52-df13-48f0-8afd-b65788ca63a7 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:19.553096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1215/2431] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:37:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:37:19.558752 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8391b315-ec1b-4c1a-bdbf-a5819a6aa44d None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:19.580713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8391b315-ec1b-4c1a-bdbf-a5819a6aa44d None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:19.586477 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554949a3-2cdb-408a-b124-1c9d5431aaf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:19.587470 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554949a3-2cdb-408a-b124-1c9d5431aaf6 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:19.587951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554949a3-2cdb-408a-b124-1c9d5431aaf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:19.588526 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554949a3-2cdb-408a-b124-1c9d5431aaf6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:19.589600 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554949a3-2cdb-408a-b124-1c9d5431aaf6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:19.593246 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554949a3-2cdb-408a-b124-1c9d5431aaf6 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:19.642846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1217/2432] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:19 2022] POST /identity/v3/projects => generated 371 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:19.643656 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba925e52-df13-48f0-8afd-b65788ca63a7 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:19.653471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba925e52-df13-48f0-8afd-b65788ca63a7 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:19.747532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3509cc7b-c096-451f-9811-c370d63c225a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:19.747532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3509cc7b-c096-451f-9811-c370d63c225a admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:19.747532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3509cc7b-c096-451f-9811-c370d63c225a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:19.747532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3509cc7b-c096-451f-9811-c370d63c225a admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:19.748031 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3509cc7b-c096-451f-9811-c370d63c225a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:19.748031 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3509cc7b-c096-451f-9811-c370d63c225a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:19.778293 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1216/2433] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:19 2022] POST /identity/v3/users => generated 286 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:37:19.782756 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554949a3-2cdb-408a-b124-1c9d5431aaf6 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:19.799789 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554949a3-2cdb-408a-b124-1c9d5431aaf6 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:19.806618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c18e288-cad6-47db-8f9c-53d82df2eab0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:19.807344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c18e288-cad6-47db-8f9c-53d82df2eab0 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:19.807704 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c18e288-cad6-47db-8f9c-53d82df2eab0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:19.807991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c18e288-cad6-47db-8f9c-53d82df2eab0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:19.808674 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c18e288-cad6-47db-8f9c-53d82df2eab0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:19.810419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c18e288-cad6-47db-8f9c-53d82df2eab0 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:19.821821 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1218/2434] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:19 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:37:19.826767 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3509cc7b-c096-451f-9811-c370d63c225a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:19.839569 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3509cc7b-c096-451f-9811-c370d63c225a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:19.841248 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c18e288-cad6-47db-8f9c-53d82df2eab0 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:19.843125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-6c18e288-cad6-47db-8f9c-53d82df2eab0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:19.845186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d306233f-93df-4dca-ac2b-d7d972f88829 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:19.845869 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d306233f-93df-4dca-ac2b-d7d972f88829 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:19.846212 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d306233f-93df-4dca-ac2b-d7d972f88829 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:19.846749 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d306233f-93df-4dca-ac2b-d7d972f88829 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:19.847768 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d306233f-93df-4dca-ac2b-d7d972f88829 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:19.849396 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d306233f-93df-4dca-ac2b-d7d972f88829 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:19.862190 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1217/2435] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:19 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:19.867045 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d306233f-93df-4dca-ac2b-d7d972f88829 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:19.879185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d306233f-93df-4dca-ac2b-d7d972f88829 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:19.885180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421f3cc6-302b-4e0e-8e51-581d2f493358 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:19.886096 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421f3cc6-302b-4e0e-8e51-581d2f493358 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:19.886437 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421f3cc6-302b-4e0e-8e51-581d2f493358 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:19.886743 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421f3cc6-302b-4e0e-8e51-581d2f493358 admin admin] PATH_INFO: `/v3/projects/b678fee2618f45a7b5689c765150f26e/users/1978295d4e254c51817b926ae91317e8/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:19.918441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421f3cc6-302b-4e0e-8e51-581d2f493358 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b678fee2618f45a7b5689c765150f26e, user_id=1978295d4e254c51817b926ae91317e8, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:19.921894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421f3cc6-302b-4e0e-8e51-581d2f493358 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:19.939004 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1218/2436] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:19 2022] PUT /identity/v3/projects/b678fee2618f45a7b5689c765150f26e/users/1978295d4e254c51817b926ae91317e8/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:37:19.951233 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feeed16e-7a9d-4895-8988-1b2a938f9f22 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:19.951581 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feeed16e-7a9d-4895-8988-1b2a938f9f22 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:19.951935 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feeed16e-7a9d-4895-8988-1b2a938f9f22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:19.955164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c18e288-cad6-47db-8f9c-53d82df2eab0 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:19.984574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-feeed16e-7a9d-4895-8988-1b2a938f9f22 None None] MFA Rules not processed for user `1978295d4e254c51817b926ae91317e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:19.986249 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-feeed16e-7a9d-4895-8988-1b2a938f9f22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:20.028216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a39d561-6e69-417d-a798-d7f61fd078b2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '436deb3d60c54a799700986519b2ef68', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:20.029329 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a39d561-6e69-417d-a798-d7f61fd078b2 service placement] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:20.029676 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a39d561-6e69-417d-a798-d7f61fd078b2 service placement] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:20.030019 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a39d561-6e69-417d-a798-d7f61fd078b2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:20.032413 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7a39d561-6e69-417d-a798-d7f61fd078b2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:20.037921 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:20.046382 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a39d561-6e69-417d-a798-d7f61fd078b2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:20.048269 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a39d561-6e69-417d-a798-d7f61fd078b2 service placement] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:20.059812 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1219/2437] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:19 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:37:20.068552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1219/2438] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:19 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:37:20.069733 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a39d561-6e69-417d-a798-d7f61fd078b2 service placement] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:20.079493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=92,keystone:DELETE=6,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:20.142681 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a39d561-6e69-417d-a798-d7f61fd078b2 service placement] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:20.147267 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690bcd9a-f2c3-4454-90e2-163a5563da26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:20.147881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690bcd9a-f2c3-4454-90e2-163a5563da26 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:20.148159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690bcd9a-f2c3-4454-90e2-163a5563da26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:20.148595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690bcd9a-f2c3-4454-90e2-163a5563da26 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:20.149707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690bcd9a-f2c3-4454-90e2-163a5563da26 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:20.151717 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690bcd9a-f2c3-4454-90e2-163a5563da26 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:20.182218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1220/2439] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:20 2022] POST /identity/v3/projects => generated 371 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:20.183290 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feeed16e-7a9d-4895-8988-1b2a938f9f22 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:20.200053 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feeed16e-7a9d-4895-8988-1b2a938f9f22 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:20.276430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11765eb7-6824-4683-9673-90e11df7e29a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:20.277085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11765eb7-6824-4683-9673-90e11df7e29a admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:20.277426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11765eb7-6824-4683-9673-90e11df7e29a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:20.277706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11765eb7-6824-4683-9673-90e11df7e29a admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:20.278350 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11765eb7-6824-4683-9673-90e11df7e29a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:20.280581 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11765eb7-6824-4683-9673-90e11df7e29a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:20.304781 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1220/2440] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:20 2022] POST /identity/v3/users => generated 292 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:37:20.309826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690bcd9a-f2c3-4454-90e2-163a5563da26 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:20.320847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690bcd9a-f2c3-4454-90e2-163a5563da26 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:20.325671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7516b61c-e55d-4ee9-84da-4f36c7d1ec3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:20.326355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7516b61c-e55d-4ee9-84da-4f36c7d1ec3d admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:20.326629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7516b61c-e55d-4ee9-84da-4f36c7d1ec3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:20.326898 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7516b61c-e55d-4ee9-84da-4f36c7d1ec3d admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:20.327538 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7516b61c-e55d-4ee9-84da-4f36c7d1ec3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:20.329174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7516b61c-e55d-4ee9-84da-4f36c7d1ec3d admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:20.340341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1221/2441] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:20.344476 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11765eb7-6824-4683-9673-90e11df7e29a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:20.355915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11765eb7-6824-4683-9673-90e11df7e29a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:20.360680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab68139-bc17-42c7-bb3e-c44a649a1c69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:20.361404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab68139-bc17-42c7-bb3e-c44a649a1c69 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:20.361737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab68139-bc17-42c7-bb3e-c44a649a1c69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:20.362013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab68139-bc17-42c7-bb3e-c44a649a1c69 admin admin] PATH_INFO: `/v3/projects/7bcf8b9118714111a55681cb26f0be34/users/e4500971858a4d73ab855e30a4e69357/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:20.381619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab68139-bc17-42c7-bb3e-c44a649a1c69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7bcf8b9118714111a55681cb26f0be34, user_id=e4500971858a4d73ab855e30a4e69357, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:20.385587 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab68139-bc17-42c7-bb3e-c44a649a1c69 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:20.400751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1221/2442] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:20 2022] PUT /identity/v3/projects/7bcf8b9118714111a55681cb26f0be34/users/e4500971858a4d73ab855e30a4e69357/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:20.405071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7516b61c-e55d-4ee9-84da-4f36c7d1ec3d admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:20.416118 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7516b61c-e55d-4ee9-84da-4f36c7d1ec3d admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:20.476476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1bc186-d9ae-4613-b695-3d4a974f6792 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:20.477760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1bc186-d9ae-4613-b695-3d4a974f6792 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:20.478098 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1bc186-d9ae-4613-b695-3d4a974f6792 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:20.478370 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1bc186-d9ae-4613-b695-3d4a974f6792 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:20.479014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1bc186-d9ae-4613-b695-3d4a974f6792 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:20.480651 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1bc186-d9ae-4613-b695-3d4a974f6792 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:20.491148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1222/2443] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:20 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:20.496029 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab68139-bc17-42c7-bb3e-c44a649a1c69 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:20.506749 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab68139-bc17-42c7-bb3e-c44a649a1c69 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:20.513652 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1eb3280-aadd-4717-834b-7cd9136f64be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:20.514384 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eb3280-aadd-4717-834b-7cd9136f64be admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:20.514591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eb3280-aadd-4717-834b-7cd9136f64be admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:20.514861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eb3280-aadd-4717-834b-7cd9136f64be admin admin] PATH_INFO: `/v3/projects/7bcf8b9118714111a55681cb26f0be34/users/e4500971858a4d73ab855e30a4e69357/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:20.516862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1eb3280-aadd-4717-834b-7cd9136f64be admin admin] RBAC: Authorizing `identity:create_grant(project_id=7bcf8b9118714111a55681cb26f0be34, user_id=e4500971858a4d73ab855e30a4e69357, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:20.519609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1eb3280-aadd-4717-834b-7cd9136f64be admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:20.532515 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1222/2444] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:20 2022] PUT /identity/v3/projects/7bcf8b9118714111a55681cb26f0be34/users/e4500971858a4d73ab855e30a4e69357/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:20.544069 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e462dcf-4d6c-46c9-981b-a910a148dfd0 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:20.544428 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e462dcf-4d6c-46c9-981b-a910a148dfd0 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:20.544893 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e462dcf-4d6c-46c9-981b-a910a148dfd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:20.583157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-0e462dcf-4d6c-46c9-981b-a910a148dfd0 None None] MFA Rules not processed for user `e4500971858a4d73ab855e30a4e69357`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:20.584795 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0e462dcf-4d6c-46c9-981b-a910a148dfd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:20.672245 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:20.677368 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1223/2445] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:20 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:20.688435 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1eb3280-aadd-4717-834b-7cd9136f64be admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:20.690178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-e1eb3280-aadd-4717-834b-7cd9136f64be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:20.760832 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1eb3280-aadd-4717-834b-7cd9136f64be admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:20.764827 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70a8cef-33ec-45de-9b45-17a830767082 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:20.765443 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70a8cef-33ec-45de-9b45-17a830767082 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:20.765717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70a8cef-33ec-45de-9b45-17a830767082 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:20.766038 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70a8cef-33ec-45de-9b45-17a830767082 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:20.779260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70a8cef-33ec-45de-9b45-17a830767082 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:20.780925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70a8cef-33ec-45de-9b45-17a830767082 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:20.794508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1223/2446] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:20 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:21.243623 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e462dcf-4d6c-46c9-981b-a910a148dfd0 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:21.325590 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e462dcf-4d6c-46c9-981b-a910a148dfd0 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:21.330253 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba39cb3-6566-4835-bb25-28343044bdde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:21.330920 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba39cb3-6566-4835-bb25-28343044bdde admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:21.331315 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba39cb3-6566-4835-bb25-28343044bdde admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:21.331726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba39cb3-6566-4835-bb25-28343044bdde admin admin] PATH_INFO: `/v3/users/1978295d4e254c51817b926ae91317e8` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:21.332935 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba39cb3-6566-4835-bb25-28343044bdde admin admin] RBAC: Authorizing `identity:delete_user(user_id=1978295d4e254c51817b926ae91317e8)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:21.334874 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba39cb3-6566-4835-bb25-28343044bdde admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:21.392028 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2ba39cb3-6566-4835-bb25-28343044bdde admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1978295d4e254c51817b926ae91317e8'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:21.393026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2ba39cb3-6566-4835-bb25-28343044bdde admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1978295d4e254c51817b926ae91317e8'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:21.393641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2ba39cb3-6566-4835-bb25-28343044bdde admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1978295d4e254c51817b926ae91317e8'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:21.404209 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2ba39cb3-6566-4835-bb25-28343044bdde admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1978295d4e254c51817b926ae91317e8'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:21.420256 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2ba39cb3-6566-4835-bb25-28343044bdde admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1978295d4e254c51817b926ae91317e8'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:21.434119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1224/2447] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:21 2022] DELETE /identity/v3/users/1978295d4e254c51817b926ae91317e8 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:37:21.438716 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8703c4ae-984a-4cf4-ab08-f9c7555aea42 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:21.438999 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8703c4ae-984a-4cf4-ab08-f9c7555aea42 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:21.439274 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8703c4ae-984a-4cf4-ab08-f9c7555aea42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:21.471445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-8703c4ae-984a-4cf4-ab08-f9c7555aea42 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:21.472877 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-8703c4ae-984a-4cf4-ab08-f9c7555aea42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:21.513785 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:21.542804 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1224/2448] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:37:21.552968 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba39cb3-6566-4835-bb25-28343044bdde admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:21.554492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-2ba39cb3-6566-4835-bb25-28343044bdde admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:21.631342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba39cb3-6566-4835-bb25-28343044bdde admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:21.635904 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba861dd-73e4-4b0c-a808-cf604ff925bd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:21.636607 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba861dd-73e4-4b0c-a808-cf604ff925bd service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:21.636891 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba861dd-73e4-4b0c-a808-cf604ff925bd service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:21.637159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba861dd-73e4-4b0c-a808-cf604ff925bd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:21.647536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba861dd-73e4-4b0c-a808-cf604ff925bd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:21.649083 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba861dd-73e4-4b0c-a808-cf604ff925bd service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:21.660629 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1225/2449] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:21.998510 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8703c4ae-984a-4cf4-ab08-f9c7555aea42 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:22.000656 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-8703c4ae-984a-4cf4-ab08-f9c7555aea42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:22.014968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8703c4ae-984a-4cf4-ab08-f9c7555aea42 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:22.021052 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:22.021706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:22.021985 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:22.022258 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10 admin admin] PATH_INFO: `/v3/projects/b678fee2618f45a7b5689c765150f26e` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:22.023407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b678fee2618f45a7b5689c765150f26e)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:22.025507 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:22.092762 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b678fee2618f45a7b5689c765150f26e', 'request_id': 'req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:22.101237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b678fee2618f45a7b5689c765150f26e', 'request_id': 'req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:22.108276 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b678fee2618f45a7b5689c765150f26e', 'request_id': 'req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:22.123718 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10 admin admin] The token cache is being invalidate because project b678fee2618f45a7b5689c765150f26e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:22.124016 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:22.124723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:22.126253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1225/2450] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:21 2022] DELETE /identity/v3/projects/b678fee2618f45a7b5689c765150f26e => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:22.130301 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba861dd-73e4-4b0c-a808-cf604ff925bd service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:22.141374 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba861dd-73e4-4b0c-a808-cf604ff925bd service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:22.167356 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:22.216369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2632be-6b01-498c-a3dd-755e3d441b88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:22.217204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2632be-6b01-498c-a3dd-755e3d441b88 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:22.217517 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2632be-6b01-498c-a3dd-755e3d441b88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:22.217801 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2632be-6b01-498c-a3dd-755e3d441b88 admin admin] PATH_INFO: `/v3/users/e4500971858a4d73ab855e30a4e69357` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:22.219070 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2632be-6b01-498c-a3dd-755e3d441b88 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4500971858a4d73ab855e30a4e69357)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:22.221081 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2632be-6b01-498c-a3dd-755e3d441b88 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:22.273780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-dd2632be-6b01-498c-a3dd-755e3d441b88 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4500971858a4d73ab855e30a4e69357'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:22.274641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-dd2632be-6b01-498c-a3dd-755e3d441b88 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4500971858a4d73ab855e30a4e69357'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:22.275044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-dd2632be-6b01-498c-a3dd-755e3d441b88 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4500971858a4d73ab855e30a4e69357'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:22.282734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-dd2632be-6b01-498c-a3dd-755e3d441b88 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4500971858a4d73ab855e30a4e69357'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:22.298319 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-dd2632be-6b01-498c-a3dd-755e3d441b88 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4500971858a4d73ab855e30a4e69357'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:22.311419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1226/2451] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:22 2022] DELETE /identity/v3/users/e4500971858a4d73ab855e30a4e69357 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:22.708851 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:22.710674 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:22.729950 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a2e80d-8a12-4504-bfd9-8da7fb74cd10 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:22.829850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8db73f4-d6db-4300-94ab-62d67c5ec46e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:22.830508 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8db73f4-d6db-4300-94ab-62d67c5ec46e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:22.830673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8db73f4-d6db-4300-94ab-62d67c5ec46e admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:22.830941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8db73f4-d6db-4300-94ab-62d67c5ec46e admin admin] PATH_INFO: `/v3/projects/7bcf8b9118714111a55681cb26f0be34` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:22.834153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8db73f4-d6db-4300-94ab-62d67c5ec46e admin admin] RBAC: Authorizing `identity:delete_project(project_id=7bcf8b9118714111a55681cb26f0be34)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:22.836984 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8db73f4-d6db-4300-94ab-62d67c5ec46e admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:22.925077 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e8db73f4-d6db-4300-94ab-62d67c5ec46e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7bcf8b9118714111a55681cb26f0be34', 'request_id': 'req-e8db73f4-d6db-4300-94ab-62d67c5ec46e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:22.935959 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e8db73f4-d6db-4300-94ab-62d67c5ec46e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7bcf8b9118714111a55681cb26f0be34', 'request_id': 'req-e8db73f4-d6db-4300-94ab-62d67c5ec46e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:22.946882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e8db73f4-d6db-4300-94ab-62d67c5ec46e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7bcf8b9118714111a55681cb26f0be34', 'request_id': 'req-e8db73f4-d6db-4300-94ab-62d67c5ec46e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:22.952774 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:22.977487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e8db73f4-d6db-4300-94ab-62d67c5ec46e admin admin] The token cache is being invalidate because project 7bcf8b9118714111a55681cb26f0be34 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:22.977795 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e8db73f4-d6db-4300-94ab-62d67c5ec46e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:22.979421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e8db73f4-d6db-4300-94ab-62d67c5ec46e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:22.986523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1226/2452] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:22 2022] DELETE /identity/v3/projects/7bcf8b9118714111a55681cb26f0be34 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:22.988196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cc2be0-e609-474f-a0ae-495db4f9802e None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:22.988781 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cc2be0-e609-474f-a0ae-495db4f9802e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:22.989272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cc2be0-e609-474f-a0ae-495db4f9802e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:23.026197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-32cc2be0-e609-474f-a0ae-495db4f9802e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:23.029637 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-32cc2be0-e609-474f-a0ae-495db4f9802e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:23.159100 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1227/2453] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:23.159573 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eabedd3-250f-4bad-84ac-7d77039bb81f None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:23.160459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eabedd3-250f-4bad-84ac-7d77039bb81f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:23.160459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eabedd3-250f-4bad-84ac-7d77039bb81f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:23.201751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-7eabedd3-250f-4bad-84ac-7d77039bb81f None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:23.203093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7eabedd3-250f-4bad-84ac-7d77039bb81f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:23.222068 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1227/2454] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:37:23.223394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cc2be0-e609-474f-a0ae-495db4f9802e None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:23.235460 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cc2be0-e609-474f-a0ae-495db4f9802e None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:23.248548 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df5720e-612d-4e60-aed6-8edaef3e63cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:23.249335 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df5720e-612d-4e60-aed6-8edaef3e63cd admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:23.249862 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df5720e-612d-4e60-aed6-8edaef3e63cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:23.250341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df5720e-612d-4e60-aed6-8edaef3e63cd admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:23.251275 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df5720e-612d-4e60-aed6-8edaef3e63cd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:37:23.251893 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df5720e-612d-4e60-aed6-8edaef3e63cd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:23.253950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df5720e-612d-4e60-aed6-8edaef3e63cd admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:23.270916 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1228/2455] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:37:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:37:23.272026 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eabedd3-250f-4bad-84ac-7d77039bb81f None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:23.283578 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eabedd3-250f-4bad-84ac-7d77039bb81f None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:23.291403 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db250a72-7fe4-4101-917d-04d48d9e0c53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:23.292289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db250a72-7fe4-4101-917d-04d48d9e0c53 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:23.292775 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db250a72-7fe4-4101-917d-04d48d9e0c53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:23.293291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db250a72-7fe4-4101-917d-04d48d9e0c53 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:23.294085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db250a72-7fe4-4101-917d-04d48d9e0c53 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:23.296276 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db250a72-7fe4-4101-917d-04d48d9e0c53 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:23.335855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1228/2456] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:23 2022] POST /identity/v3/projects => generated 367 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:37:23.340761 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df5720e-612d-4e60-aed6-8edaef3e63cd admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:23.364624 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df5720e-612d-4e60-aed6-8edaef3e63cd admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:23.387878 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db250a72-7fe4-4101-917d-04d48d9e0c53 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:23.389961 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-db250a72-7fe4-4101-917d-04d48d9e0c53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:23.404817 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:23.465879 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef60ab89-3d3b-4ae0-910e-57facf8091b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:23.466585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef60ab89-3d3b-4ae0-910e-57facf8091b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:23.467049 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef60ab89-3d3b-4ae0-910e-57facf8091b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:23.467492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef60ab89-3d3b-4ae0-910e-57facf8091b1 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:23.468132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef60ab89-3d3b-4ae0-910e-57facf8091b1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:23.470163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef60ab89-3d3b-4ae0-910e-57facf8091b1 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:23.485873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db250a72-7fe4-4101-917d-04d48d9e0c53 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:23.490909 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb34e6e2-5620-48d8-a37e-c2b933eb1732 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:23.491544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb34e6e2-5620-48d8-a37e-c2b933eb1732 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:23.492526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb34e6e2-5620-48d8-a37e-c2b933eb1732 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:23.492811 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb34e6e2-5620-48d8-a37e-c2b933eb1732 admin admin] PATH_INFO: `/v3/users/cab40292db0d4a74b1fd59c3539e81e3` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:23.494580 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb34e6e2-5620-48d8-a37e-c2b933eb1732 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cab40292db0d4a74b1fd59c3539e81e3)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:23.496566 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb34e6e2-5620-48d8-a37e-c2b933eb1732 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:23.501909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1229/2457] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:23 2022] POST /identity/v3/users => generated 284 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:23.502957 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef60ab89-3d3b-4ae0-910e-57facf8091b1 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:23.512952 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef60ab89-3d3b-4ae0-910e-57facf8091b1 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:23.518261 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46d669a-ba9c-490d-814c-1ed7539fb03e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:23.518884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46d669a-ba9c-490d-814c-1ed7539fb03e admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:23.519169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46d669a-ba9c-490d-814c-1ed7539fb03e admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:23.519440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46d669a-ba9c-490d-814c-1ed7539fb03e admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:23.520085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46d669a-ba9c-490d-814c-1ed7539fb03e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:23.521966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46d669a-ba9c-490d-814c-1ed7539fb03e admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:23.535168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1230/2458] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:23 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:23.539996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46d669a-ba9c-490d-814c-1ed7539fb03e admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:23.548335 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-eb34e6e2-5620-48d8-a37e-c2b933eb1732 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cab40292db0d4a74b1fd59c3539e81e3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:23.550735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46d669a-ba9c-490d-814c-1ed7539fb03e admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:23.556066 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d7db38-a5c7-48a1-bd22-c1a240d7c663 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:23.556811 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-eb34e6e2-5620-48d8-a37e-c2b933eb1732 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cab40292db0d4a74b1fd59c3539e81e3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:23.556939 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d7db38-a5c7-48a1-bd22-c1a240d7c663 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:23.557199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d7db38-a5c7-48a1-bd22-c1a240d7c663 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:23.557476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d7db38-a5c7-48a1-bd22-c1a240d7c663 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:23.558115 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d7db38-a5c7-48a1-bd22-c1a240d7c663 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:23.559718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d7db38-a5c7-48a1-bd22-c1a240d7c663 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:23.564259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-eb34e6e2-5620-48d8-a37e-c2b933eb1732 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cab40292db0d4a74b1fd59c3539e81e3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:23.570349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1231/2459] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:23 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:37:23.574350 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d7db38-a5c7-48a1-bd22-c1a240d7c663 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:23.577813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-eb34e6e2-5620-48d8-a37e-c2b933eb1732 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cab40292db0d4a74b1fd59c3539e81e3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:23.578521 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-eb34e6e2-5620-48d8-a37e-c2b933eb1732 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cab40292db0d4a74b1fd59c3539e81e3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:23.584071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d7db38-a5c7-48a1-bd22-c1a240d7c663 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:23.585266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1229/2460] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:23 2022] DELETE /identity/v3/users/cab40292db0d4a74b1fd59c3539e81e3 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:23.662550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd0ccca-c9cc-4992-ab70-e1fa5e62c9e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:23.663237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd0ccca-c9cc-4992-ab70-e1fa5e62c9e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:23.663522 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd0ccca-c9cc-4992-ab70-e1fa5e62c9e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:23.663853 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd0ccca-c9cc-4992-ab70-e1fa5e62c9e2 admin admin] PATH_INFO: `/v3/projects/2c7648119a8148b2aa6f2d604279b95c/users/2767255b0f1f4ab28874a5c11e8e738f/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:23.680442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd0ccca-c9cc-4992-ab70-e1fa5e62c9e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2c7648119a8148b2aa6f2d604279b95c, user_id=2767255b0f1f4ab28874a5c11e8e738f, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:23.683127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd0ccca-c9cc-4992-ab70-e1fa5e62c9e2 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:23.698456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1232/2461] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:23 2022] PUT /identity/v3/projects/2c7648119a8148b2aa6f2d604279b95c/users/2767255b0f1f4ab28874a5c11e8e738f/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:37:23.709885 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c7e61e-233e-46af-b3cf-3daadf056879 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:23.710209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c7e61e-233e-46af-b3cf-3daadf056879 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:23.710448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c7e61e-233e-46af-b3cf-3daadf056879 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:23.744832 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-b3c7e61e-233e-46af-b3cf-3daadf056879 None None] MFA Rules not processed for user `2767255b0f1f4ab28874a5c11e8e738f`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:23.746382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b3c7e61e-233e-46af-b3cf-3daadf056879 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:23.769410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd0ccca-c9cc-4992-ab70-e1fa5e62c9e2 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:23.771110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-fdd0ccca-c9cc-4992-ab70-e1fa5e62c9e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:23.789765 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:23.831275 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1230/2462] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:23 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:23.832339 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c7e61e-233e-46af-b3cf-3daadf056879 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:23.833759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b3c7e61e-233e-46af-b3cf-3daadf056879 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:23.852889 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd0ccca-c9cc-4992-ab70-e1fa5e62c9e2 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:23.858261 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5d7a8e-6c2d-4865-9f47-48389125b162 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:23.858889 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5d7a8e-6c2d-4865-9f47-48389125b162 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:23.859169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5d7a8e-6c2d-4865-9f47-48389125b162 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:23.859445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5d7a8e-6c2d-4865-9f47-48389125b162 admin admin] PATH_INFO: `/v3/projects/9cf9a725cf7e42ea939bee84113b12ee` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:23.860593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5d7a8e-6c2d-4865-9f47-48389125b162 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9cf9a725cf7e42ea939bee84113b12ee)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:23.862529 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5d7a8e-6c2d-4865-9f47-48389125b162 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:23.886564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:23.903737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c7e61e-233e-46af-b3cf-3daadf056879 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:23.912566 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-7f5d7a8e-6c2d-4865-9f47-48389125b162 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9cf9a725cf7e42ea939bee84113b12ee', 'request_id': 'req-7f5d7a8e-6c2d-4865-9f47-48389125b162'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:23.919860 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-7f5d7a8e-6c2d-4865-9f47-48389125b162 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9cf9a725cf7e42ea939bee84113b12ee', 'request_id': 'req-7f5d7a8e-6c2d-4865-9f47-48389125b162'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:23.926415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-7f5d7a8e-6c2d-4865-9f47-48389125b162 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9cf9a725cf7e42ea939bee84113b12ee', 'request_id': 'req-7f5d7a8e-6c2d-4865-9f47-48389125b162'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:23.939053 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-7f5d7a8e-6c2d-4865-9f47-48389125b162 admin admin] The token cache is being invalidate because project 9cf9a725cf7e42ea939bee84113b12ee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:23.939348 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-7f5d7a8e-6c2d-4865-9f47-48389125b162 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:23.939996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-7f5d7a8e-6c2d-4865-9f47-48389125b162 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:23.942248 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1233/2463] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:23 2022] DELETE /identity/v3/projects/9cf9a725cf7e42ea939bee84113b12ee => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:37:23.946832 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5d7a8e-6c2d-4865-9f47-48389125b162 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:23.948272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7f5d7a8e-6c2d-4865-9f47-48389125b162 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:23.968120 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0ecae0-8f2c-4669-afff-f22174fe1eaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:23.969064 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0ecae0-8f2c-4669-afff-f22174fe1eaf admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:23.969064 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0ecae0-8f2c-4669-afff-f22174fe1eaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:23.969313 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0ecae0-8f2c-4669-afff-f22174fe1eaf admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:23.969889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0ecae0-8f2c-4669-afff-f22174fe1eaf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:23.971786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0ecae0-8f2c-4669-afff-f22174fe1eaf admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:24.004724 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1231/2464] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:23 2022] POST /identity/v3/projects => generated 369 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:24.009369 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0ecae0-8f2c-4669-afff-f22174fe1eaf admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:24.011132 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-4b0ecae0-8f2c-4669-afff-f22174fe1eaf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:24.021196 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5d7a8e-6c2d-4865-9f47-48389125b162 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:24.085675 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0ecae0-8f2c-4669-afff-f22174fe1eaf admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:24.089072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b35ae9-28ed-4a38-9960-ea2d29b05f71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:24.090092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b35ae9-28ed-4a38-9960-ea2d29b05f71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:24.090598 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b35ae9-28ed-4a38-9960-ea2d29b05f71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:24.091068 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b35ae9-28ed-4a38-9960-ea2d29b05f71 admin admin] PATH_INFO: `/v3/users/724b08c95d8d4386b0ca7c40c5fdb696` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:24.092902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b35ae9-28ed-4a38-9960-ea2d29b05f71 admin admin] RBAC: Authorizing `identity:delete_user(user_id=724b08c95d8d4386b0ca7c40c5fdb696)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:24.094026 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9196e690-a05a-4500-99f5-10b35ce63e6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:24.094956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9196e690-a05a-4500-99f5-10b35ce63e6f admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:24.095408 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9196e690-a05a-4500-99f5-10b35ce63e6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:24.095859 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9196e690-a05a-4500-99f5-10b35ce63e6f admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:24.096363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b35ae9-28ed-4a38-9960-ea2d29b05f71 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:24.096986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9196e690-a05a-4500-99f5-10b35ce63e6f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:24.100285 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9196e690-a05a-4500-99f5-10b35ce63e6f admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:24.114641 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:24.129261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1232/2465] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:24 2022] POST /identity/v3/users => generated 291 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:37:24.132537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9196e690-a05a-4500-99f5-10b35ce63e6f admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:24.132537 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9196e690-a05a-4500-99f5-10b35ce63e6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:24.145905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9196e690-a05a-4500-99f5-10b35ce63e6f admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:24.152090 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d1b35ae9-28ed-4a38-9960-ea2d29b05f71 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '724b08c95d8d4386b0ca7c40c5fdb696'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:24.152782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d1b35ae9-28ed-4a38-9960-ea2d29b05f71 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '724b08c95d8d4386b0ca7c40c5fdb696'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:24.152915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85ca088-7d9e-4389-86ff-a50723c20ee3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:24.153443 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d1b35ae9-28ed-4a38-9960-ea2d29b05f71 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '724b08c95d8d4386b0ca7c40c5fdb696'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:24.153558 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85ca088-7d9e-4389-86ff-a50723c20ee3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:24.153783 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85ca088-7d9e-4389-86ff-a50723c20ee3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:24.154308 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85ca088-7d9e-4389-86ff-a50723c20ee3 admin admin] PATH_INFO: `/v3/users/53a0c6b231364be59b9c57493d3d821d` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:24.155631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85ca088-7d9e-4389-86ff-a50723c20ee3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=53a0c6b231364be59b9c57493d3d821d)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:24.157591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85ca088-7d9e-4389-86ff-a50723c20ee3 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:24.161634 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d1b35ae9-28ed-4a38-9960-ea2d29b05f71 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '724b08c95d8d4386b0ca7c40c5fdb696'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:24.166469 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:24.176404 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d1b35ae9-28ed-4a38-9960-ea2d29b05f71 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '724b08c95d8d4386b0ca7c40c5fdb696'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:24.192357 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1234/2466] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:23 2022] DELETE /identity/v3/users/724b08c95d8d4386b0ca7c40c5fdb696 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:37:24.196012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b35ae9-28ed-4a38-9960-ea2d29b05f71 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:24.196012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d1b35ae9-28ed-4a38-9960-ea2d29b05f71 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:24.212003 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b85ca088-7d9e-4389-86ff-a50723c20ee3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '53a0c6b231364be59b9c57493d3d821d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:24.222405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b85ca088-7d9e-4389-86ff-a50723c20ee3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '53a0c6b231364be59b9c57493d3d821d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:24.233358 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b85ca088-7d9e-4389-86ff-a50723c20ee3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '53a0c6b231364be59b9c57493d3d821d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:24.252147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b85ca088-7d9e-4389-86ff-a50723c20ee3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '53a0c6b231364be59b9c57493d3d821d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:24.252990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-b85ca088-7d9e-4389-86ff-a50723c20ee3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '53a0c6b231364be59b9c57493d3d821d'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:24.260657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1233/2467] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:24 2022] DELETE /identity/v3/users/53a0c6b231364be59b9c57493d3d821d => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:24.264971 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d759994-1e7b-411e-823f-e06fda574daa None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:24.265334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d759994-1e7b-411e-823f-e06fda574daa None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:24.265545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d759994-1e7b-411e-823f-e06fda574daa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:24.271335 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b35ae9-28ed-4a38-9960-ea2d29b05f71 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:24.289573 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-5d759994-1e7b-411e-823f-e06fda574daa None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:24.290995 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5d759994-1e7b-411e-823f-e06fda574daa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:24.341564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736d39ed-5111-4e18-9313-57896eb05ee8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:24.342790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736d39ed-5111-4e18-9313-57896eb05ee8 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:24.343074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736d39ed-5111-4e18-9313-57896eb05ee8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:24.343346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736d39ed-5111-4e18-9313-57896eb05ee8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:24.344098 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736d39ed-5111-4e18-9313-57896eb05ee8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:24.346120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736d39ed-5111-4e18-9313-57896eb05ee8 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:24.359074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1235/2468] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:24 2022] GET /identity/v3/roles => generated 1420 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:24.360353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1234/2469] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:24.363122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736d39ed-5111-4e18-9313-57896eb05ee8 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:24.364877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-736d39ed-5111-4e18-9313-57896eb05ee8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:24.370589 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d759994-1e7b-411e-823f-e06fda574daa None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:24.372566 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5d759994-1e7b-411e-823f-e06fda574daa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:24.378852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736d39ed-5111-4e18-9313-57896eb05ee8 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:24.394564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3ff195-9534-4fcd-adf5-8cd3146bac68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:24.395504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3ff195-9534-4fcd-adf5-8cd3146bac68 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:24.396237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3ff195-9534-4fcd-adf5-8cd3146bac68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:24.396664 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3ff195-9534-4fcd-adf5-8cd3146bac68 admin admin] PATH_INFO: `/v3/projects/1b00634588204b41ac2df7546c1e195c/users/fc075caa5fd3409997be065b91b5690e/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:24.424619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3ff195-9534-4fcd-adf5-8cd3146bac68 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b00634588204b41ac2df7546c1e195c, user_id=fc075caa5fd3409997be065b91b5690e, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:24.427651 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3ff195-9534-4fcd-adf5-8cd3146bac68 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:24.440260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d759994-1e7b-411e-823f-e06fda574daa None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:24.442013 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1236/2470] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:24 2022] PUT /identity/v3/projects/1b00634588204b41ac2df7546c1e195c/users/fc075caa5fd3409997be065b91b5690e/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:37:24.442895 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3ff195-9534-4fcd-adf5-8cd3146bac68 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:24.444540 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-bd3ff195-9534-4fcd-adf5-8cd3146bac68 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:24.466214 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:24.501290 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26218e85-721d-40e2-9421-fbedc419cb89 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:24.501927 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26218e85-721d-40e2-9421-fbedc419cb89 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:24.502201 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26218e85-721d-40e2-9421-fbedc419cb89 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:24.502471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26218e85-721d-40e2-9421-fbedc419cb89 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:24.512910 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:DELETE=2,keystone:SELECT=97,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:24.522933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3ff195-9534-4fcd-adf5-8cd3146bac68 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:24.532207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0abd43-9d74-4a62-90a7-d565dd64b96f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:24.532886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0abd43-9d74-4a62-90a7-d565dd64b96f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:24.533167 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0abd43-9d74-4a62-90a7-d565dd64b96f admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:24.533441 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0abd43-9d74-4a62-90a7-d565dd64b96f admin admin] PATH_INFO: `/v3/projects/941e341c4f564c77a33a6944b6d7bf63` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:24.534643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0abd43-9d74-4a62-90a7-d565dd64b96f admin admin] RBAC: Authorizing `identity:delete_project(project_id=941e341c4f564c77a33a6944b6d7bf63)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:24.536554 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0abd43-9d74-4a62-90a7-d565dd64b96f admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:24.573070 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26218e85-721d-40e2-9421-fbedc419cb89 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:24.574579 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26218e85-721d-40e2-9421-fbedc419cb89 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:24.585071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1235/2471] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:37:24.589255 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26218e85-721d-40e2-9421-fbedc419cb89 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:24.594742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-ad0abd43-9d74-4a62-90a7-d565dd64b96f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '941e341c4f564c77a33a6944b6d7bf63', 'request_id': 'req-ad0abd43-9d74-4a62-90a7-d565dd64b96f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:24.599432 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26218e85-721d-40e2-9421-fbedc419cb89 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:24.607298 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-ad0abd43-9d74-4a62-90a7-d565dd64b96f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '941e341c4f564c77a33a6944b6d7bf63', 'request_id': 'req-ad0abd43-9d74-4a62-90a7-d565dd64b96f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:24.618088 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-ad0abd43-9d74-4a62-90a7-d565dd64b96f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '941e341c4f564c77a33a6944b6d7bf63', 'request_id': 'req-ad0abd43-9d74-4a62-90a7-d565dd64b96f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:24.637340 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-ad0abd43-9d74-4a62-90a7-d565dd64b96f admin admin] The token cache is being invalidate because project 941e341c4f564c77a33a6944b6d7bf63 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:24.637632 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-ad0abd43-9d74-4a62-90a7-d565dd64b96f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:24.638333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-ad0abd43-9d74-4a62-90a7-d565dd64b96f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:24.640068 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1237/2472] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:24 2022] DELETE /identity/v3/projects/941e341c4f564c77a33a6944b6d7bf63 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:24.687144 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b318b1bc-05bf-4ed9-ae6e-63e67ac975e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:24.687890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b318b1bc-05bf-4ed9-ae6e-63e67ac975e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:24.688258 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b318b1bc-05bf-4ed9-ae6e-63e67ac975e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:24.688674 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b318b1bc-05bf-4ed9-ae6e-63e67ac975e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:24.689572 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b318b1bc-05bf-4ed9-ae6e-63e67ac975e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:24.691538 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b318b1bc-05bf-4ed9-ae6e-63e67ac975e7 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:24.710052 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0abd43-9d74-4a62-90a7-d565dd64b96f admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:24.712884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ad0abd43-9d74-4a62-90a7-d565dd64b96f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:24.714566 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1236/2473] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:24 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:24.730612 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0abd43-9d74-4a62-90a7-d565dd64b96f admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:24.737071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910cdb64-6a00-4699-982b-efbb74e4e15b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:24.738077 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910cdb64-6a00-4699-982b-efbb74e4e15b admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:24.738466 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910cdb64-6a00-4699-982b-efbb74e4e15b admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:24.738852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910cdb64-6a00-4699-982b-efbb74e4e15b admin admin] PATH_INFO: `/v3/projects/1b00634588204b41ac2df7546c1e195c/users/fc075caa5fd3409997be065b91b5690e/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:24.741332 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910cdb64-6a00-4699-982b-efbb74e4e15b admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b00634588204b41ac2df7546c1e195c, user_id=fc075caa5fd3409997be065b91b5690e, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:24.746238 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910cdb64-6a00-4699-982b-efbb74e4e15b admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:24.775684 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1238/2474] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:24 2022] PUT /identity/v3/projects/1b00634588204b41ac2df7546c1e195c/users/fc075caa5fd3409997be065b91b5690e/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:24.777926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a612a90c-6f04-402a-b43b-5ae0fba65c33 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:24.778362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a612a90c-6f04-402a-b43b-5ae0fba65c33 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:24.778743 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a612a90c-6f04-402a-b43b-5ae0fba65c33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:24.823789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-a612a90c-6f04-402a-b43b-5ae0fba65c33 None None] MFA Rules not processed for user `fc075caa5fd3409997be065b91b5690e`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:24.825307 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a612a90c-6f04-402a-b43b-5ae0fba65c33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:24.930579 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:24.944383 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1237/2475] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:24 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:37:24.952364 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910cdb64-6a00-4699-982b-efbb74e4e15b admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:24.954079 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-910cdb64-6a00-4699-982b-efbb74e4e15b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:25.006039 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a612a90c-6f04-402a-b43b-5ae0fba65c33 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:25.007957 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a612a90c-6f04-402a-b43b-5ae0fba65c33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:25.016687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910cdb64-6a00-4699-982b-efbb74e4e15b admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:25.020995 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c86bd1a-e086-4bf4-8420-174772cb5bf8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:25.021893 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c86bd1a-e086-4bf4-8420-174772cb5bf8 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:25.022559 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c86bd1a-e086-4bf4-8420-174772cb5bf8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:25.022856 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c86bd1a-e086-4bf4-8420-174772cb5bf8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:25.034861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c86bd1a-e086-4bf4-8420-174772cb5bf8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:25.036657 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c86bd1a-e086-4bf4-8420-174772cb5bf8 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:25.048624 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1239/2476] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:24 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:25.083613 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a612a90c-6f04-402a-b43b-5ae0fba65c33 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:25.089712 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda6de1f-fa74-4b87-850c-b3c362501193 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:25.090197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda6de1f-fa74-4b87-850c-b3c362501193 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:25.090469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda6de1f-fa74-4b87-850c-b3c362501193 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:25.090737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda6de1f-fa74-4b87-850c-b3c362501193 admin admin] PATH_INFO: `/v3/projects/47c2da5b0ed14916bbed7796f8575d10` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:25.091955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda6de1f-fa74-4b87-850c-b3c362501193 admin admin] RBAC: Authorizing `identity:delete_project(project_id=47c2da5b0ed14916bbed7796f8575d10)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:25.093916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda6de1f-fa74-4b87-850c-b3c362501193 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:25.158455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-eda6de1f-fa74-4b87-850c-b3c362501193 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '47c2da5b0ed14916bbed7796f8575d10', 'request_id': 'req-eda6de1f-fa74-4b87-850c-b3c362501193'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:25.166486 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-eda6de1f-fa74-4b87-850c-b3c362501193 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '47c2da5b0ed14916bbed7796f8575d10', 'request_id': 'req-eda6de1f-fa74-4b87-850c-b3c362501193'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:25.172925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-eda6de1f-fa74-4b87-850c-b3c362501193 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '47c2da5b0ed14916bbed7796f8575d10', 'request_id': 'req-eda6de1f-fa74-4b87-850c-b3c362501193'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:25.187006 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-eda6de1f-fa74-4b87-850c-b3c362501193 admin admin] The token cache is being invalidate because project 47c2da5b0ed14916bbed7796f8575d10 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:25.187306 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-eda6de1f-fa74-4b87-850c-b3c362501193 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:25.187977 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-eda6de1f-fa74-4b87-850c-b3c362501193 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:25.189684 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1238/2477] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:25 2022] DELETE /identity/v3/projects/47c2da5b0ed14916bbed7796f8575d10 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:37:25.197251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458795f1-dc48-4169-a59c-d720ac0c8a66 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:25.197540 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458795f1-dc48-4169-a59c-d720ac0c8a66 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:25.197817 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458795f1-dc48-4169-a59c-d720ac0c8a66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:25.228154 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-458795f1-dc48-4169-a59c-d720ac0c8a66 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:25.229414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-458795f1-dc48-4169-a59c-d720ac0c8a66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:25.298607 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1240/2478] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:25.372884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f172c23c-276b-4925-a697-13fd408be428 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:25.373361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f172c23c-276b-4925-a697-13fd408be428 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:25.373484 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f172c23c-276b-4925-a697-13fd408be428 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:25.399065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-f172c23c-276b-4925-a697-13fd408be428 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:25.400277 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f172c23c-276b-4925-a697-13fd408be428 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:25.412523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1239/2479] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:25.413437 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458795f1-dc48-4169-a59c-d720ac0c8a66 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:25.424914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458795f1-dc48-4169-a59c-d720ac0c8a66 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:25.432124 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8de1174-7654-4e7b-b2cd-ab3dd61dd00c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:25.432788 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8de1174-7654-4e7b-b2cd-ab3dd61dd00c admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:25.433276 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8de1174-7654-4e7b-b2cd-ab3dd61dd00c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:25.433571 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8de1174-7654-4e7b-b2cd-ab3dd61dd00c admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:25.434230 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8de1174-7654-4e7b-b2cd-ab3dd61dd00c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:37:25.434536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8de1174-7654-4e7b-b2cd-ab3dd61dd00c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:25.436157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8de1174-7654-4e7b-b2cd-ab3dd61dd00c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:25.457466 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1241/2480] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:37:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:37:25.462820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f172c23c-276b-4925-a697-13fd408be428 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:25.483920 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f172c23c-276b-4925-a697-13fd408be428 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:25.488960 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9c8b01-c9eb-4fa0-9b53-baf81c0a405d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:25.489603 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9c8b01-c9eb-4fa0-9b53-baf81c0a405d admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:25.489880 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9c8b01-c9eb-4fa0-9b53-baf81c0a405d admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:25.490152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9c8b01-c9eb-4fa0-9b53-baf81c0a405d admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:25.490786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9c8b01-c9eb-4fa0-9b53-baf81c0a405d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:25.493483 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9c8b01-c9eb-4fa0-9b53-baf81c0a405d admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:25.525520 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1240/2481] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:25 2022] POST /identity/v3/projects => generated 373 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:37:25.530712 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8de1174-7654-4e7b-b2cd-ab3dd61dd00c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:25.540651 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:25.551520 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8de1174-7654-4e7b-b2cd-ab3dd61dd00c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:25.629847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bdaf76-416a-4d0b-ac3b-c3ecfc5246f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:25.630472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bdaf76-416a-4d0b-ac3b-c3ecfc5246f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:25.630758 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bdaf76-416a-4d0b-ac3b-c3ecfc5246f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:25.631036 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bdaf76-416a-4d0b-ac3b-c3ecfc5246f9 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:25.631672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bdaf76-416a-4d0b-ac3b-c3ecfc5246f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:25.634611 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bdaf76-416a-4d0b-ac3b-c3ecfc5246f9 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:25.663083 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1242/2482] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:25 2022] POST /identity/v3/users => generated 287 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:25.664499 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9c8b01-c9eb-4fa0-9b53-baf81c0a405d admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:25.675755 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9c8b01-c9eb-4fa0-9b53-baf81c0a405d admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:25.680477 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54a7120-feeb-4859-a571-cee24df8a5dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:25.681219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54a7120-feeb-4859-a571-cee24df8a5dc admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:25.681486 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54a7120-feeb-4859-a571-cee24df8a5dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:25.681855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54a7120-feeb-4859-a571-cee24df8a5dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:25.682380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54a7120-feeb-4859-a571-cee24df8a5dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:25.683943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54a7120-feeb-4859-a571-cee24df8a5dc admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:25.694304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1241/2483] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:25.698658 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bdaf76-416a-4d0b-ac3b-c3ecfc5246f9 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:25.709358 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bdaf76-416a-4d0b-ac3b-c3ecfc5246f9 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:25.714906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f9fc36-f42b-4764-bee1-8b5442a81f1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:25.715527 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f9fc36-f42b-4764-bee1-8b5442a81f1d admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:25.715805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f9fc36-f42b-4764-bee1-8b5442a81f1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:25.716073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f9fc36-f42b-4764-bee1-8b5442a81f1d admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:25.716749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f9fc36-f42b-4764-bee1-8b5442a81f1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:25.718322 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f9fc36-f42b-4764-bee1-8b5442a81f1d admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:25.730175 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1243/2484] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:25 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:25.734832 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54a7120-feeb-4859-a571-cee24df8a5dc admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:25.745228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54a7120-feeb-4859-a571-cee24df8a5dc admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:25.750453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7fe0e1-d524-4d60-9e18-0aa5f0b654f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:25.751206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7fe0e1-d524-4d60-9e18-0aa5f0b654f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:25.751539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7fe0e1-d524-4d60-9e18-0aa5f0b654f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:25.751944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7fe0e1-d524-4d60-9e18-0aa5f0b654f1 admin admin] PATH_INFO: `/v3/projects/bf0ffb6098014feb89d5d6f61832b03b/users/33fb70b5ab4942a38771709eeb3a5789/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:25.768294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7fe0e1-d524-4d60-9e18-0aa5f0b654f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bf0ffb6098014feb89d5d6f61832b03b, user_id=33fb70b5ab4942a38771709eeb3a5789, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:25.771195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7fe0e1-d524-4d60-9e18-0aa5f0b654f1 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:25.790935 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1242/2485] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:25 2022] PUT /identity/v3/projects/bf0ffb6098014feb89d5d6f61832b03b/users/33fb70b5ab4942a38771709eeb3a5789/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:25.793310 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5744bb29-a696-4f99-b211-851b3cceb6a8 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:25.793711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5744bb29-a696-4f99-b211-851b3cceb6a8 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:25.794067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5744bb29-a696-4f99-b211-851b3cceb6a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:25.820257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-5744bb29-a696-4f99-b211-851b3cceb6a8 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:25.821971 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-5744bb29-a696-4f99-b211-851b3cceb6a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:25.899368 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1244/2486] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:25.909821 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7fe0e1-d524-4d60-9e18-0aa5f0b654f1 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:25.911650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-4f7fe0e1-d524-4d60-9e18-0aa5f0b654f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:25.951408 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:25.990927 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7fe0e1-d524-4d60-9e18-0aa5f0b654f1 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:25.995441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98398b49-e0d9-4b64-8a60-ccedc4dab047 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:25.996057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98398b49-e0d9-4b64-8a60-ccedc4dab047 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:25.996353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98398b49-e0d9-4b64-8a60-ccedc4dab047 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:25.996632 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98398b49-e0d9-4b64-8a60-ccedc4dab047 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:26.006381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98398b49-e0d9-4b64-8a60-ccedc4dab047 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:26.008022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98398b49-e0d9-4b64-8a60-ccedc4dab047 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:26.018083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1243/2487] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:26.619960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5744bb29-a696-4f99-b211-851b3cceb6a8 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:26.621467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-5744bb29-a696-4f99-b211-851b3cceb6a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:26.636059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5744bb29-a696-4f99-b211-851b3cceb6a8 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:26.641393 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e396f29-38e2-4d9a-ba55-26aa436c9a41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:26.642133 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e396f29-38e2-4d9a-ba55-26aa436c9a41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:26.642416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e396f29-38e2-4d9a-ba55-26aa436c9a41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:26.642570 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e396f29-38e2-4d9a-ba55-26aa436c9a41 admin admin] PATH_INFO: `/v3/users/2767255b0f1f4ab28874a5c11e8e738f` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:26.643720 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e396f29-38e2-4d9a-ba55-26aa436c9a41 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2767255b0f1f4ab28874a5c11e8e738f)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:26.645694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e396f29-38e2-4d9a-ba55-26aa436c9a41 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:26.792900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8e396f29-38e2-4d9a-ba55-26aa436c9a41 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2767255b0f1f4ab28874a5c11e8e738f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:26.794050 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8e396f29-38e2-4d9a-ba55-26aa436c9a41 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2767255b0f1f4ab28874a5c11e8e738f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:26.795459 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8e396f29-38e2-4d9a-ba55-26aa436c9a41 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2767255b0f1f4ab28874a5c11e8e738f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:26.810803 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8e396f29-38e2-4d9a-ba55-26aa436c9a41 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2767255b0f1f4ab28874a5c11e8e738f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:26.830878 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8e396f29-38e2-4d9a-ba55-26aa436c9a41 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2767255b0f1f4ab28874a5c11e8e738f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:26.856870 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:26.858593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1245/2488] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:26 2022] DELETE /identity/v3/users/2767255b0f1f4ab28874a5c11e8e738f => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:26.865067 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bdbda3-240a-467c-9585-e624cecc0250 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:26.865678 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bdbda3-240a-467c-9585-e624cecc0250 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:26.865892 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bdbda3-240a-467c-9585-e624cecc0250 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:26.902805 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-96bdbda3-240a-467c-9585-e624cecc0250 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:26.904475 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-96bdbda3-240a-467c-9585-e624cecc0250 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:27.067214 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1244/2489] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:27.088437 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e396f29-38e2-4d9a-ba55-26aa436c9a41 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:27.090850 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8e396f29-38e2-4d9a-ba55-26aa436c9a41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:27.229559 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e396f29-38e2-4d9a-ba55-26aa436c9a41 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:27.233515 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37c415f-4cb4-4473-b051-66bd3c7585af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:27.234137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37c415f-4cb4-4473-b051-66bd3c7585af service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:27.234535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37c415f-4cb4-4473-b051-66bd3c7585af service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:27.234688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37c415f-4cb4-4473-b051-66bd3c7585af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:27.246149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37c415f-4cb4-4473-b051-66bd3c7585af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:27.247644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37c415f-4cb4-4473-b051-66bd3c7585af service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:27.259367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1246/2490] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:27.706005 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96bdbda3-240a-467c-9585-e624cecc0250 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:27.708497 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-96bdbda3-240a-467c-9585-e624cecc0250 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:27.735223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96bdbda3-240a-467c-9585-e624cecc0250 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:27.746061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b36bf2d-7fac-4204-8516-c64a3b612569 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:27.748638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b36bf2d-7fac-4204-8516-c64a3b612569 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:27.749500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b36bf2d-7fac-4204-8516-c64a3b612569 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:27.750480 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b36bf2d-7fac-4204-8516-c64a3b612569 admin admin] PATH_INFO: `/v3/projects/2c7648119a8148b2aa6f2d604279b95c` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:27.758464 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b36bf2d-7fac-4204-8516-c64a3b612569 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2c7648119a8148b2aa6f2d604279b95c)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:27.759746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b36bf2d-7fac-4204-8516-c64a3b612569 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:27.832717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-0b36bf2d-7fac-4204-8516-c64a3b612569 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2c7648119a8148b2aa6f2d604279b95c', 'request_id': 'req-0b36bf2d-7fac-4204-8516-c64a3b612569'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:27.842550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-0b36bf2d-7fac-4204-8516-c64a3b612569 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2c7648119a8148b2aa6f2d604279b95c', 'request_id': 'req-0b36bf2d-7fac-4204-8516-c64a3b612569'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:27.850460 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-0b36bf2d-7fac-4204-8516-c64a3b612569 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2c7648119a8148b2aa6f2d604279b95c', 'request_id': 'req-0b36bf2d-7fac-4204-8516-c64a3b612569'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:27.881526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-0b36bf2d-7fac-4204-8516-c64a3b612569 admin admin] The token cache is being invalidate because project 2c7648119a8148b2aa6f2d604279b95c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:27.883159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-0b36bf2d-7fac-4204-8516-c64a3b612569 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:27.884123 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-0b36bf2d-7fac-4204-8516-c64a3b612569 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:27.885999 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1245/2491] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:27 2022] DELETE /identity/v3/projects/2c7648119a8148b2aa6f2d604279b95c => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:27.890403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37c415f-4cb4-4473-b051-66bd3c7585af service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:27.903368 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37c415f-4cb4-4473-b051-66bd3c7585af service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:28.023742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c538999c-ba60-4a28-adbd-3eba0f3cba20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:28.024660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c538999c-ba60-4a28-adbd-3eba0f3cba20 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:28.024936 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c538999c-ba60-4a28-adbd-3eba0f3cba20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:28.025377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c538999c-ba60-4a28-adbd-3eba0f3cba20 admin admin] PATH_INFO: `/v3/users/fc075caa5fd3409997be065b91b5690e` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:28.026912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c538999c-ba60-4a28-adbd-3eba0f3cba20 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc075caa5fd3409997be065b91b5690e)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:28.030002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c538999c-ba60-4a28-adbd-3eba0f3cba20 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:28.104954 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c538999c-ba60-4a28-adbd-3eba0f3cba20 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc075caa5fd3409997be065b91b5690e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:28.105580 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c538999c-ba60-4a28-adbd-3eba0f3cba20 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc075caa5fd3409997be065b91b5690e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:28.106286 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c538999c-ba60-4a28-adbd-3eba0f3cba20 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc075caa5fd3409997be065b91b5690e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:28.121368 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c538999c-ba60-4a28-adbd-3eba0f3cba20 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc075caa5fd3409997be065b91b5690e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:28.137987 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c538999c-ba60-4a28-adbd-3eba0f3cba20 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc075caa5fd3409997be065b91b5690e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:28.152202 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1247/2492] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:27 2022] DELETE /identity/v3/users/fc075caa5fd3409997be065b91b5690e => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:28.656287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b36bf2d-7fac-4204-8516-c64a3b612569 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:28.659156 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-0b36bf2d-7fac-4204-8516-c64a3b612569 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:28.674198 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b36bf2d-7fac-4204-8516-c64a3b612569 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:28.762080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:28.805523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623ac7db-2079-4553-993c-582f5285ce54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:28.806375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623ac7db-2079-4553-993c-582f5285ce54 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:28.806846 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623ac7db-2079-4553-993c-582f5285ce54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:28.807123 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623ac7db-2079-4553-993c-582f5285ce54 admin admin] PATH_INFO: `/v3/projects/1b00634588204b41ac2df7546c1e195c` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:28.808432 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623ac7db-2079-4553-993c-582f5285ce54 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1b00634588204b41ac2df7546c1e195c)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:28.810361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623ac7db-2079-4553-993c-582f5285ce54 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:28.905531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-623ac7db-2079-4553-993c-582f5285ce54 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1b00634588204b41ac2df7546c1e195c', 'request_id': 'req-623ac7db-2079-4553-993c-582f5285ce54'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:28.915425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-623ac7db-2079-4553-993c-582f5285ce54 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1b00634588204b41ac2df7546c1e195c', 'request_id': 'req-623ac7db-2079-4553-993c-582f5285ce54'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:28.925193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-623ac7db-2079-4553-993c-582f5285ce54 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1b00634588204b41ac2df7546c1e195c', 'request_id': 'req-623ac7db-2079-4553-993c-582f5285ce54'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:28.948820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-623ac7db-2079-4553-993c-582f5285ce54 admin admin] The token cache is being invalidate because project 1b00634588204b41ac2df7546c1e195c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:28.948970 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-623ac7db-2079-4553-993c-582f5285ce54 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:28.950103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-623ac7db-2079-4553-993c-582f5285ce54 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:28.954281 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1246/2493] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:28 2022] DELETE /identity/v3/projects/1b00634588204b41ac2df7546c1e195c => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:28.959806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7cadac-e048-43d7-b90e-37057dad28fe None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:28.960966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7cadac-e048-43d7-b90e-37057dad28fe None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:28.961385 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7cadac-e048-43d7-b90e-37057dad28fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:28.989499 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-0b7cadac-e048-43d7-b90e-37057dad28fe None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:28.990741 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0b7cadac-e048-43d7-b90e-37057dad28fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:29.013119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:29.096513 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1248/2494] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:37:29.106090 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add96425-c0b7-4864-bf56-02424480075d None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:29.106370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add96425-c0b7-4864-bf56-02424480075d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:29.106642 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add96425-c0b7-4864-bf56-02424480075d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:29.129039 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-add96425-c0b7-4864-bf56-02424480075d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:29.130194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-add96425-c0b7-4864-bf56-02424480075d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:29.138857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1247/2495] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:29.146091 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7cadac-e048-43d7-b90e-37057dad28fe None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:29.160886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7cadac-e048-43d7-b90e-37057dad28fe None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:29.171366 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07c83fe-f68c-49c8-baa2-b0bfb7e4cb7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:29.172029 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07c83fe-f68c-49c8-baa2-b0bfb7e4cb7c admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:29.172537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07c83fe-f68c-49c8-baa2-b0bfb7e4cb7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:29.173109 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07c83fe-f68c-49c8-baa2-b0bfb7e4cb7c admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:29.174085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07c83fe-f68c-49c8-baa2-b0bfb7e4cb7c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:37:29.175351 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07c83fe-f68c-49c8-baa2-b0bfb7e4cb7c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:29.177288 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07c83fe-f68c-49c8-baa2-b0bfb7e4cb7c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:29.193007 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1249/2496] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:37:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:37:29.198806 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add96425-c0b7-4864-bf56-02424480075d None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:29.217151 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add96425-c0b7-4864-bf56-02424480075d None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:29.222688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68d22b7-8a2d-486a-a6b4-4c7274b4f834 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:29.223645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68d22b7-8a2d-486a-a6b4-4c7274b4f834 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:29.224285 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68d22b7-8a2d-486a-a6b4-4c7274b4f834 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:29.224796 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68d22b7-8a2d-486a-a6b4-4c7274b4f834 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:29.225733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68d22b7-8a2d-486a-a6b4-4c7274b4f834 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:29.228726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68d22b7-8a2d-486a-a6b4-4c7274b4f834 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:29.286063 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1248/2497] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:29 2022] POST /identity/v3/projects => generated 379 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:29.291448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07c83fe-f68c-49c8-baa2-b0bfb7e4cb7c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:29.307280 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07c83fe-f68c-49c8-baa2-b0bfb7e4cb7c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:29.428980 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba29707-997e-485a-8f71-cef5ef204de5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:29.430254 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba29707-997e-485a-8f71-cef5ef204de5 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:29.430860 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba29707-997e-485a-8f71-cef5ef204de5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:29.431294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba29707-997e-485a-8f71-cef5ef204de5 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:29.432138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba29707-997e-485a-8f71-cef5ef204de5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:29.434218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba29707-997e-485a-8f71-cef5ef204de5 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:29.471936 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1250/2498] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:29 2022] POST /identity/v3/users => generated 290 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:29.477797 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68d22b7-8a2d-486a-a6b4-4c7274b4f834 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:29.491651 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68d22b7-8a2d-486a-a6b4-4c7274b4f834 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:29.502986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21d741d-0d1f-4539-ab92-33c08ca9ad27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:29.503630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21d741d-0d1f-4539-ab92-33c08ca9ad27 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:29.503901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21d741d-0d1f-4539-ab92-33c08ca9ad27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:29.504366 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21d741d-0d1f-4539-ab92-33c08ca9ad27 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:29.505193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21d741d-0d1f-4539-ab92-33c08ca9ad27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:29.506789 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21d741d-0d1f-4539-ab92-33c08ca9ad27 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:29.517687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1249/2499] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:29 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:29.521993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba29707-997e-485a-8f71-cef5ef204de5 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:29.533511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba29707-997e-485a-8f71-cef5ef204de5 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:29.541812 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e656c0ca-02f5-41f2-980a-323afbe2114d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:29.542846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e656c0ca-02f5-41f2-980a-323afbe2114d admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:29.543346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e656c0ca-02f5-41f2-980a-323afbe2114d admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:29.543833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e656c0ca-02f5-41f2-980a-323afbe2114d admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:29.544917 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e656c0ca-02f5-41f2-980a-323afbe2114d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:29.547112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e656c0ca-02f5-41f2-980a-323afbe2114d admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:29.575489 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1251/2500] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:29 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:29.578421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21d741d-0d1f-4539-ab92-33c08ca9ad27 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:29.589914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21d741d-0d1f-4539-ab92-33c08ca9ad27 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:29.595768 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114f7822-9d8d-4f3b-8934-4dc952cb6a0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:29.596538 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114f7822-9d8d-4f3b-8934-4dc952cb6a0f admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:29.596819 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114f7822-9d8d-4f3b-8934-4dc952cb6a0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:29.597097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114f7822-9d8d-4f3b-8934-4dc952cb6a0f admin admin] PATH_INFO: `/v3/projects/5923af4cdabf4237805ad0ae68a9fffc/users/71497e10ea8947b88c863b81efef0c94/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:29.620522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114f7822-9d8d-4f3b-8934-4dc952cb6a0f admin admin] RBAC: Authorizing `identity:create_grant(project_id=5923af4cdabf4237805ad0ae68a9fffc, user_id=71497e10ea8947b88c863b81efef0c94, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:29.623258 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114f7822-9d8d-4f3b-8934-4dc952cb6a0f admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:29.641974 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1250/2501] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:29 2022] PUT /identity/v3/projects/5923af4cdabf4237805ad0ae68a9fffc/users/71497e10ea8947b88c863b81efef0c94/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:37:29.656017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3712ddad-4ca7-4d5c-9e73-210bc570db50 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:29.656353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3712ddad-4ca7-4d5c-9e73-210bc570db50 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:29.656719 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3712ddad-4ca7-4d5c-9e73-210bc570db50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:29.691930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-3712ddad-4ca7-4d5c-9e73-210bc570db50 None None] MFA Rules not processed for user `71497e10ea8947b88c863b81efef0c94`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:29.693334 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-3712ddad-4ca7-4d5c-9e73-210bc570db50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:29.776383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1252/2502] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:29 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:29.777765 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114f7822-9d8d-4f3b-8934-4dc952cb6a0f admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:29.846279 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114f7822-9d8d-4f3b-8934-4dc952cb6a0f admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:29.852709 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f0da6b-3b4f-4635-9f5f-62f00a6dbaa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:29.853432 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f0da6b-3b4f-4635-9f5f-62f00a6dbaa0 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:29.853815 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f0da6b-3b4f-4635-9f5f-62f00a6dbaa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:29.854173 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f0da6b-3b4f-4635-9f5f-62f00a6dbaa0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:29.855007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f0da6b-3b4f-4635-9f5f-62f00a6dbaa0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:29.858199 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f0da6b-3b4f-4635-9f5f-62f00a6dbaa0 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:29.884461 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:29.896300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1251/2503] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:29 2022] POST /identity/v3/projects => generated 377 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:29.897382 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3712ddad-4ca7-4d5c-9e73-210bc570db50 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:29.914957 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3712ddad-4ca7-4d5c-9e73-210bc570db50 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:29.926186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:29.993053 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724dc61b-7bec-4ae1-8e70-5b43dd85c9cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:29.993802 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724dc61b-7bec-4ae1-8e70-5b43dd85c9cf admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:29.994103 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724dc61b-7bec-4ae1-8e70-5b43dd85c9cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:29.994417 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724dc61b-7bec-4ae1-8e70-5b43dd85c9cf admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:29.995169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724dc61b-7bec-4ae1-8e70-5b43dd85c9cf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:29.997339 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724dc61b-7bec-4ae1-8e70-5b43dd85c9cf admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:30.020391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1253/2504] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:29 2022] POST /identity/v3/users => generated 289 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:37:30.025120 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f0da6b-3b4f-4635-9f5f-62f00a6dbaa0 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:30.040073 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f0da6b-3b4f-4635-9f5f-62f00a6dbaa0 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:30.046767 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308c9334-9bc7-40a0-8cb6-e389ccaaa82f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:30.047657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308c9334-9bc7-40a0-8cb6-e389ccaaa82f admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:30.048127 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308c9334-9bc7-40a0-8cb6-e389ccaaa82f admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:30.048585 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308c9334-9bc7-40a0-8cb6-e389ccaaa82f admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:30.049605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308c9334-9bc7-40a0-8cb6-e389ccaaa82f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:30.051622 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308c9334-9bc7-40a0-8cb6-e389ccaaa82f admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:30.065351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1252/2505] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:30 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:30.070345 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724dc61b-7bec-4ae1-8e70-5b43dd85c9cf admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:30.082738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724dc61b-7bec-4ae1-8e70-5b43dd85c9cf admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:30.089587 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6243d4c2-75cc-4c4a-8ecf-316b2aeee106 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:30.090378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6243d4c2-75cc-4c4a-8ecf-316b2aeee106 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:30.090708 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6243d4c2-75cc-4c4a-8ecf-316b2aeee106 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:30.091066 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6243d4c2-75cc-4c4a-8ecf-316b2aeee106 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:30.092069 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6243d4c2-75cc-4c4a-8ecf-316b2aeee106 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:30.094029 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6243d4c2-75cc-4c4a-8ecf-316b2aeee106 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:30.116450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1254/2506] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:30 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:30.121298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308c9334-9bc7-40a0-8cb6-e389ccaaa82f admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:30.132091 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308c9334-9bc7-40a0-8cb6-e389ccaaa82f admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:30.138045 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0580c48-af9b-4fdf-a191-a6dc1557596c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:30.138864 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0580c48-af9b-4fdf-a191-a6dc1557596c admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:30.139246 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0580c48-af9b-4fdf-a191-a6dc1557596c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:30.139737 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0580c48-af9b-4fdf-a191-a6dc1557596c admin admin] PATH_INFO: `/v3/projects/12a4370ffdcc4eb29a5da4a0084c2141/users/8c0b1ec3acef4952baf7adbc46b812a7/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:30.156545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0580c48-af9b-4fdf-a191-a6dc1557596c admin admin] RBAC: Authorizing `identity:create_grant(project_id=12a4370ffdcc4eb29a5da4a0084c2141, user_id=8c0b1ec3acef4952baf7adbc46b812a7, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:30.159640 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0580c48-af9b-4fdf-a191-a6dc1557596c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:30.177236 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1253/2507] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:30 2022] PUT /identity/v3/projects/12a4370ffdcc4eb29a5da4a0084c2141/users/8c0b1ec3acef4952baf7adbc46b812a7/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:30.188394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315f4047-dfa5-4166-9b43-48a861161e4a None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:30.188705 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315f4047-dfa5-4166-9b43-48a861161e4a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:30.188975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315f4047-dfa5-4166-9b43-48a861161e4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:30.222739 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-315f4047-dfa5-4166-9b43-48a861161e4a None None] MFA Rules not processed for user `8c0b1ec3acef4952baf7adbc46b812a7`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:30.224114 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-315f4047-dfa5-4166-9b43-48a861161e4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:30.314049 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1255/2508] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:30 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:30.318736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0580c48-af9b-4fdf-a191-a6dc1557596c admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:30.406987 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0580c48-af9b-4fdf-a191-a6dc1557596c admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:30.416383 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da6054f-08c6-4718-9b0d-8bdb2c2f4fef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:30.417134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da6054f-08c6-4718-9b0d-8bdb2c2f4fef admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:30.417526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da6054f-08c6-4718-9b0d-8bdb2c2f4fef admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:30.420327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da6054f-08c6-4718-9b0d-8bdb2c2f4fef admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:30.420327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da6054f-08c6-4718-9b0d-8bdb2c2f4fef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:30.421312 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da6054f-08c6-4718-9b0d-8bdb2c2f4fef admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:30.457955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1254/2509] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:30 2022] POST /identity/v3/projects => generated 377 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:37:30.459132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315f4047-dfa5-4166-9b43-48a861161e4a None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:30.474589 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315f4047-dfa5-4166-9b43-48a861161e4a None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:30.575445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9621eab-a5fc-4068-b39b-9e79c70f6b94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:30.580685 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9621eab-a5fc-4068-b39b-9e79c70f6b94 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:30.581400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9621eab-a5fc-4068-b39b-9e79c70f6b94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:30.581861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9621eab-a5fc-4068-b39b-9e79c70f6b94 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:30.582747 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9621eab-a5fc-4068-b39b-9e79c70f6b94 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:30.584989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9621eab-a5fc-4068-b39b-9e79c70f6b94 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:30.614656 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1256/2510] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:30 2022] POST /identity/v3/users => generated 295 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:37:30.619267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da6054f-08c6-4718-9b0d-8bdb2c2f4fef admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:30.644967 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da6054f-08c6-4718-9b0d-8bdb2c2f4fef admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:30.653070 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1245d6-3cde-4104-9e52-9c0ce9c09c7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:30.655329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1245d6-3cde-4104-9e52-9c0ce9c09c7d admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:30.655673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1245d6-3cde-4104-9e52-9c0ce9c09c7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:30.655909 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1245d6-3cde-4104-9e52-9c0ce9c09c7d admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:30.657632 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1245d6-3cde-4104-9e52-9c0ce9c09c7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:30.659335 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1245d6-3cde-4104-9e52-9c0ce9c09c7d admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:30.688981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1255/2511] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:30 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:37:30.689986 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9621eab-a5fc-4068-b39b-9e79c70f6b94 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:30.705840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9621eab-a5fc-4068-b39b-9e79c70f6b94 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:30.713577 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096e1f76-bdd1-49e1-bbc0-088a8ef979d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:30.714780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096e1f76-bdd1-49e1-bbc0-088a8ef979d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:30.715309 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096e1f76-bdd1-49e1-bbc0-088a8ef979d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:30.715754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096e1f76-bdd1-49e1-bbc0-088a8ef979d7 admin admin] PATH_INFO: `/v3/projects/53bc05af028c40ef83456e69e9668799/users/ee54e6f8589a4f69b9cf1428ce5c09a7/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:30.743194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:30.783872 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096e1f76-bdd1-49e1-bbc0-088a8ef979d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=53bc05af028c40ef83456e69e9668799, user_id=ee54e6f8589a4f69b9cf1428ce5c09a7, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:30.787058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096e1f76-bdd1-49e1-bbc0-088a8ef979d7 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:30.832439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1257/2512] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:30 2022] PUT /identity/v3/projects/53bc05af028c40ef83456e69e9668799/users/ee54e6f8589a4f69b9cf1428ce5c09a7/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:37:30.836028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1245d6-3cde-4104-9e52-9c0ce9c09c7d admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:30.861294 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1245d6-3cde-4104-9e52-9c0ce9c09c7d admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:30.980184 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5e83cd-9a40-489a-a8f9-a418e67a26d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:30.981017 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5e83cd-9a40-489a-a8f9-a418e67a26d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:30.981309 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5e83cd-9a40-489a-a8f9-a418e67a26d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:30.981626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5e83cd-9a40-489a-a8f9-a418e67a26d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:30.982305 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5e83cd-9a40-489a-a8f9-a418e67a26d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:30.983915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5e83cd-9a40-489a-a8f9-a418e67a26d1 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:30.999703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1256/2513] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:30 2022] GET /identity/v3/roles => generated 1420 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:31.004282 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096e1f76-bdd1-49e1-bbc0-088a8ef979d7 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:31.018668 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096e1f76-bdd1-49e1-bbc0-088a8ef979d7 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:31.032764 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994cea44-76ff-4460-8849-ff08d1bd7f61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:31.033450 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994cea44-76ff-4460-8849-ff08d1bd7f61 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:31.033771 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994cea44-76ff-4460-8849-ff08d1bd7f61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:31.033990 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994cea44-76ff-4460-8849-ff08d1bd7f61 admin admin] PATH_INFO: `/v3/projects/53bc05af028c40ef83456e69e9668799/users/ee54e6f8589a4f69b9cf1428ce5c09a7/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:31.037917 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994cea44-76ff-4460-8849-ff08d1bd7f61 admin admin] RBAC: Authorizing `identity:create_grant(project_id=53bc05af028c40ef83456e69e9668799, user_id=ee54e6f8589a4f69b9cf1428ce5c09a7, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:31.041817 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994cea44-76ff-4460-8849-ff08d1bd7f61 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:31.086520 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1258/2514] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:31 2022] PUT /identity/v3/projects/53bc05af028c40ef83456e69e9668799/users/ee54e6f8589a4f69b9cf1428ce5c09a7/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:31.090335 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8967b4-a14d-4234-95e0-2a7ee4cea2ec None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:31.108400 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8967b4-a14d-4234-95e0-2a7ee4cea2ec None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:31.108722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8967b4-a14d-4234-95e0-2a7ee4cea2ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:31.160670 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-fe8967b4-a14d-4234-95e0-2a7ee4cea2ec None None] MFA Rules not processed for user `ee54e6f8589a4f69b9cf1428ce5c09a7`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:31.163095 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-fe8967b4-a14d-4234-95e0-2a7ee4cea2ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:31.199247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:31.272590 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1257/2515] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:31 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:31.292115 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994cea44-76ff-4460-8849-ff08d1bd7f61 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:31.295062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-994cea44-76ff-4460-8849-ff08d1bd7f61 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:31.392943 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994cea44-76ff-4460-8849-ff08d1bd7f61 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:31.400118 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c0c296-8d72-49bf-9376-f0a3ab50bc02 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:31.408579 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c0c296-8d72-49bf-9376-f0a3ab50bc02 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:31.409005 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c0c296-8d72-49bf-9376-f0a3ab50bc02 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:31.409382 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c0c296-8d72-49bf-9376-f0a3ab50bc02 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:31.433711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c0c296-8d72-49bf-9376-f0a3ab50bc02 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:31.435296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c0c296-8d72-49bf-9376-f0a3ab50bc02 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:31.496109 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1259/2516] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:31 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:31.555246 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8967b4-a14d-4234-95e0-2a7ee4cea2ec None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:31.557687 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-fe8967b4-a14d-4234-95e0-2a7ee4cea2ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:31.579256 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8967b4-a14d-4234-95e0-2a7ee4cea2ec None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:31.589485 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6ab742-2f56-46c9-ad94-ae4d0ecc09b9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:31.590124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6ab742-2f56-46c9-ad94-ae4d0ecc09b9 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:31.590404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6ab742-2f56-46c9-ad94-ae4d0ecc09b9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:31.590679 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6ab742-2f56-46c9-ad94-ae4d0ecc09b9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:31.704328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6ab742-2f56-46c9-ad94-ae4d0ecc09b9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:31.705815 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6ab742-2f56-46c9-ad94-ae4d0ecc09b9 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:31.727232 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1258/2517] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:31 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:31.757182 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736cf83b-6223-45d4-9576-a05c201e7b2a None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:31.757463 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736cf83b-6223-45d4-9576-a05c201e7b2a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:31.758078 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736cf83b-6223-45d4-9576-a05c201e7b2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:31.797994 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-736cf83b-6223-45d4-9576-a05c201e7b2a None None] MFA Rules not processed for user `33fb70b5ab4942a38771709eeb3a5789`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:31.799239 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-736cf83b-6223-45d4-9576-a05c201e7b2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:31.936175 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1260/2518] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:31 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:31.941059 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6ab742-2f56-46c9-ad94-ae4d0ecc09b9 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:31.943630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-df6ab742-2f56-46c9-ad94-ae4d0ecc09b9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:32.120514 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6ab742-2f56-46c9-ad94-ae4d0ecc09b9 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:32.128731 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a17e6b-b16d-42e7-be26-a2f73cbd196f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:32.129356 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a17e6b-b16d-42e7-be26-a2f73cbd196f admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:32.129633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a17e6b-b16d-42e7-be26-a2f73cbd196f admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:32.129906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a17e6b-b16d-42e7-be26-a2f73cbd196f admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:32.130592 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a17e6b-b16d-42e7-be26-a2f73cbd196f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:32.133862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a17e6b-b16d-42e7-be26-a2f73cbd196f admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:32.175248 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1259/2519] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:31 2022] POST /identity/v3/projects => generated 373 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:32.176179 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736cf83b-6223-45d4-9576-a05c201e7b2a None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:32.187368 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736cf83b-6223-45d4-9576-a05c201e7b2a None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:32.343040 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef497257-bac1-4f64-8aee-5c814a09628a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:32.345244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef497257-bac1-4f64-8aee-5c814a09628a admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:32.345531 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef497257-bac1-4f64-8aee-5c814a09628a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:32.347149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef497257-bac1-4f64-8aee-5c814a09628a admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:32.347816 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef497257-bac1-4f64-8aee-5c814a09628a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:32.356280 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef497257-bac1-4f64-8aee-5c814a09628a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:32.392710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:32.409094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1261/2520] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:32 2022] POST /identity/v3/users => generated 293 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:32.415083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a17e6b-b16d-42e7-be26-a2f73cbd196f admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:32.427867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a17e6b-b16d-42e7-be26-a2f73cbd196f admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:32.434720 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4447ba55-bc4f-4f73-8538-108f7a580060 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:32.440412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4447ba55-bc4f-4f73-8538-108f7a580060 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:32.440412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4447ba55-bc4f-4f73-8538-108f7a580060 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:32.440412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4447ba55-bc4f-4f73-8538-108f7a580060 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:32.440412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4447ba55-bc4f-4f73-8538-108f7a580060 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:32.440412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4447ba55-bc4f-4f73-8538-108f7a580060 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:32.461076 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1260/2521] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:32 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:32.463582 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef497257-bac1-4f64-8aee-5c814a09628a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:32.476203 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef497257-bac1-4f64-8aee-5c814a09628a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:32.483531 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5c3dd0-d295-4aa9-b814-bacf4b86d10d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:32.484216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5c3dd0-d295-4aa9-b814-bacf4b86d10d admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:32.484531 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5c3dd0-d295-4aa9-b814-bacf4b86d10d admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:32.484811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5c3dd0-d295-4aa9-b814-bacf4b86d10d admin admin] PATH_INFO: `/v3/projects/3b511402d62d48d7ae28d4b36513bacb/users/8a6d28b8bb5d46c1940cda041d1c56b5/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:32.501504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5c3dd0-d295-4aa9-b814-bacf4b86d10d admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b511402d62d48d7ae28d4b36513bacb, user_id=8a6d28b8bb5d46c1940cda041d1c56b5, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:32.504209 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5c3dd0-d295-4aa9-b814-bacf4b86d10d admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:32.519413 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1262/2522] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:32 2022] PUT /identity/v3/projects/3b511402d62d48d7ae28d4b36513bacb/users/8a6d28b8bb5d46c1940cda041d1c56b5/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:32.524244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4447ba55-bc4f-4f73-8538-108f7a580060 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:32.534780 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4447ba55-bc4f-4f73-8538-108f7a580060 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:32.568102 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:32.599285 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06b0eaf-da14-4ab8-afe0-bca39473dd47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:32.599908 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b0eaf-da14-4ab8-afe0-bca39473dd47 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:32.600185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b0eaf-da14-4ab8-afe0-bca39473dd47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:32.600478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b0eaf-da14-4ab8-afe0-bca39473dd47 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:32.601278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06b0eaf-da14-4ab8-afe0-bca39473dd47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:32.602954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06b0eaf-da14-4ab8-afe0-bca39473dd47 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:32.614498 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1261/2523] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:32 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:32.619018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5c3dd0-d295-4aa9-b814-bacf4b86d10d admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:32.630551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5c3dd0-d295-4aa9-b814-bacf4b86d10d admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:32.640094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5ad282-3160-4805-bf5d-f216b5c156c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:32.641020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5ad282-3160-4805-bf5d-f216b5c156c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:32.641364 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5ad282-3160-4805-bf5d-f216b5c156c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:32.641705 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5ad282-3160-4805-bf5d-f216b5c156c0 admin admin] PATH_INFO: `/v3/projects/3b511402d62d48d7ae28d4b36513bacb/users/8a6d28b8bb5d46c1940cda041d1c56b5/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:32.644159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5ad282-3160-4805-bf5d-f216b5c156c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b511402d62d48d7ae28d4b36513bacb, user_id=8a6d28b8bb5d46c1940cda041d1c56b5, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:32.647137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5ad282-3160-4805-bf5d-f216b5c156c0 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:32.664471 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1263/2524] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:32 2022] PUT /identity/v3/projects/3b511402d62d48d7ae28d4b36513bacb/users/8a6d28b8bb5d46c1940cda041d1c56b5/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:35.416091 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06b0eaf-da14-4ab8-afe0-bca39473dd47 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:35.427437 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06b0eaf-da14-4ab8-afe0-bca39473dd47 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:35.517930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f8a43e-74ef-4a21-98fa-d6d87678478d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:35.518585 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f8a43e-74ef-4a21-98fa-d6d87678478d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:35.518866 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f8a43e-74ef-4a21-98fa-d6d87678478d admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:35.519426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f8a43e-74ef-4a21-98fa-d6d87678478d admin admin] PATH_INFO: `/v3/users/71497e10ea8947b88c863b81efef0c94` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:35.520678 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f8a43e-74ef-4a21-98fa-d6d87678478d admin admin] RBAC: Authorizing `identity:delete_user(user_id=71497e10ea8947b88c863b81efef0c94)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:35.523610 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f8a43e-74ef-4a21-98fa-d6d87678478d admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:35.590149 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-a3f8a43e-74ef-4a21-98fa-d6d87678478d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '71497e10ea8947b88c863b81efef0c94'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:35.598055 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-a3f8a43e-74ef-4a21-98fa-d6d87678478d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '71497e10ea8947b88c863b81efef0c94'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:35.607716 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-a3f8a43e-74ef-4a21-98fa-d6d87678478d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '71497e10ea8947b88c863b81efef0c94'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:35.622371 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-a3f8a43e-74ef-4a21-98fa-d6d87678478d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '71497e10ea8947b88c863b81efef0c94'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:35.623130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-a3f8a43e-74ef-4a21-98fa-d6d87678478d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '71497e10ea8947b88c863b81efef0c94'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:35.630884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1262/2525] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:35 2022] DELETE /identity/v3/users/71497e10ea8947b88c863b81efef0c94 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:35.635592 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54f5f2b-df15-4cd3-a050-58319fc08121 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:35.636157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54f5f2b-df15-4cd3-a050-58319fc08121 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:35.636157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54f5f2b-df15-4cd3-a050-58319fc08121 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:35.677816 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-a54f5f2b-df15-4cd3-a050-58319fc08121 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:35.680182 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a54f5f2b-df15-4cd3-a050-58319fc08121 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:35.763138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1264/2526] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:35.771023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f8a43e-74ef-4a21-98fa-d6d87678478d admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:35.774676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a3f8a43e-74ef-4a21-98fa-d6d87678478d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:35.899643 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:35.916118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f8a43e-74ef-4a21-98fa-d6d87678478d admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:35.919836 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c949edb-eead-4e6c-a6a6-b8f33fbd094a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:35.920601 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c949edb-eead-4e6c-a6a6-b8f33fbd094a service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:35.921007 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c949edb-eead-4e6c-a6a6-b8f33fbd094a service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:35.921346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c949edb-eead-4e6c-a6a6-b8f33fbd094a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:35.942288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c949edb-eead-4e6c-a6a6-b8f33fbd094a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:35.943967 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c949edb-eead-4e6c-a6a6-b8f33fbd094a service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:35.965393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1263/2527] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:36.809632 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54f5f2b-df15-4cd3-a050-58319fc08121 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:36.812291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a54f5f2b-df15-4cd3-a050-58319fc08121 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:36.838533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54f5f2b-df15-4cd3-a050-58319fc08121 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:36.861902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3606d14-74e7-4c7b-876e-2b596de7537e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:36.863704 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3606d14-74e7-4c7b-876e-2b596de7537e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:36.865301 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3606d14-74e7-4c7b-876e-2b596de7537e admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:36.868183 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3606d14-74e7-4c7b-876e-2b596de7537e admin admin] PATH_INFO: `/v3/projects/5923af4cdabf4237805ad0ae68a9fffc` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:36.872255 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3606d14-74e7-4c7b-876e-2b596de7537e admin admin] RBAC: Authorizing `identity:delete_project(project_id=5923af4cdabf4237805ad0ae68a9fffc)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:36.878875 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3606d14-74e7-4c7b-876e-2b596de7537e admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:37.009857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d3606d14-74e7-4c7b-876e-2b596de7537e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5923af4cdabf4237805ad0ae68a9fffc', 'request_id': 'req-d3606d14-74e7-4c7b-876e-2b596de7537e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:37.020462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d3606d14-74e7-4c7b-876e-2b596de7537e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5923af4cdabf4237805ad0ae68a9fffc', 'request_id': 'req-d3606d14-74e7-4c7b-876e-2b596de7537e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:37.031082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d3606d14-74e7-4c7b-876e-2b596de7537e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5923af4cdabf4237805ad0ae68a9fffc', 'request_id': 'req-d3606d14-74e7-4c7b-876e-2b596de7537e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:37.047759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d3606d14-74e7-4c7b-876e-2b596de7537e admin admin] The token cache is being invalidate because project 5923af4cdabf4237805ad0ae68a9fffc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:37.048266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d3606d14-74e7-4c7b-876e-2b596de7537e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:37.049928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d3606d14-74e7-4c7b-876e-2b596de7537e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:37.052226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1265/2528] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:36 2022] DELETE /identity/v3/projects/5923af4cdabf4237805ad0ae68a9fffc => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:37.056697 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c949edb-eead-4e6c-a6a6-b8f33fbd094a service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:37.072263 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c949edb-eead-4e6c-a6a6-b8f33fbd094a service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:37.241554 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdba6d8-6e91-496f-b417-06b6ec6c3e43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:37.242509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdba6d8-6e91-496f-b417-06b6ec6c3e43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:37.242910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdba6d8-6e91-496f-b417-06b6ec6c3e43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:37.243287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdba6d8-6e91-496f-b417-06b6ec6c3e43 admin admin] PATH_INFO: `/v3/users/8c0b1ec3acef4952baf7adbc46b812a7` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:37.245774 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdba6d8-6e91-496f-b417-06b6ec6c3e43 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8c0b1ec3acef4952baf7adbc46b812a7)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:37.248261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdba6d8-6e91-496f-b417-06b6ec6c3e43 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:37.326281 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-1bdba6d8-6e91-496f-b417-06b6ec6c3e43 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8c0b1ec3acef4952baf7adbc46b812a7'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:37.336839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-1bdba6d8-6e91-496f-b417-06b6ec6c3e43 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8c0b1ec3acef4952baf7adbc46b812a7'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:37.345730 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-1bdba6d8-6e91-496f-b417-06b6ec6c3e43 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8c0b1ec3acef4952baf7adbc46b812a7'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:37.369430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-1bdba6d8-6e91-496f-b417-06b6ec6c3e43 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8c0b1ec3acef4952baf7adbc46b812a7'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:37.370462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-1bdba6d8-6e91-496f-b417-06b6ec6c3e43 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8c0b1ec3acef4952baf7adbc46b812a7'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:37.386939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1264/2529] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:37 2022] DELETE /identity/v3/users/8c0b1ec3acef4952baf7adbc46b812a7 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:37:37.782251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3606d14-74e7-4c7b-876e-2b596de7537e admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:37.783923 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d3606d14-74e7-4c7b-876e-2b596de7537e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:37.818262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3606d14-74e7-4c7b-876e-2b596de7537e admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:37.897185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:INSERT=4,keystone:SELECT=89,keystone:DELETE=7 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:37.941006 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e31b9d0-9cd9-4ada-833a-fab3c874d4e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:37.942822 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e31b9d0-9cd9-4ada-833a-fab3c874d4e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:37.944687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e31b9d0-9cd9-4ada-833a-fab3c874d4e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:37.945033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e31b9d0-9cd9-4ada-833a-fab3c874d4e0 admin admin] PATH_INFO: `/v3/projects/12a4370ffdcc4eb29a5da4a0084c2141` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:37.948533 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e31b9d0-9cd9-4ada-833a-fab3c874d4e0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=12a4370ffdcc4eb29a5da4a0084c2141)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:37.950986 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e31b9d0-9cd9-4ada-833a-fab3c874d4e0 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:38.041144 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-1e31b9d0-9cd9-4ada-833a-fab3c874d4e0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '12a4370ffdcc4eb29a5da4a0084c2141', 'request_id': 'req-1e31b9d0-9cd9-4ada-833a-fab3c874d4e0'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:38.053358 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-1e31b9d0-9cd9-4ada-833a-fab3c874d4e0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '12a4370ffdcc4eb29a5da4a0084c2141', 'request_id': 'req-1e31b9d0-9cd9-4ada-833a-fab3c874d4e0'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:38.064899 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-1e31b9d0-9cd9-4ada-833a-fab3c874d4e0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '12a4370ffdcc4eb29a5da4a0084c2141', 'request_id': 'req-1e31b9d0-9cd9-4ada-833a-fab3c874d4e0'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:38.097690 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-1e31b9d0-9cd9-4ada-833a-fab3c874d4e0 admin admin] The token cache is being invalidate because project 12a4370ffdcc4eb29a5da4a0084c2141 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:38.097931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-1e31b9d0-9cd9-4ada-833a-fab3c874d4e0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:38.099072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-1e31b9d0-9cd9-4ada-833a-fab3c874d4e0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:38.102028 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1266/2530] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:37 2022] DELETE /identity/v3/projects/12a4370ffdcc4eb29a5da4a0084c2141 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:38.108607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdba6d8-6e91-496f-b417-06b6ec6c3e43 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:38.111459 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1bdba6d8-6e91-496f-b417-06b6ec6c3e43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:38.146330 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdba6d8-6e91-496f-b417-06b6ec6c3e43 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:38.239053 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c4d96d-3bd3-453f-bdf1-5eb06bd89181 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:38.239962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c4d96d-3bd3-453f-bdf1-5eb06bd89181 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:38.240428 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c4d96d-3bd3-453f-bdf1-5eb06bd89181 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:38.240883 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c4d96d-3bd3-453f-bdf1-5eb06bd89181 admin admin] PATH_INFO: `/v3/users/ee54e6f8589a4f69b9cf1428ce5c09a7` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:38.244628 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c4d96d-3bd3-453f-bdf1-5eb06bd89181 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee54e6f8589a4f69b9cf1428ce5c09a7)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:38.247180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c4d96d-3bd3-453f-bdf1-5eb06bd89181 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:38.268241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:38.361328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-73c4d96d-3bd3-453f-bdf1-5eb06bd89181 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee54e6f8589a4f69b9cf1428ce5c09a7'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:38.374956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-73c4d96d-3bd3-453f-bdf1-5eb06bd89181 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee54e6f8589a4f69b9cf1428ce5c09a7'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:38.391913 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-73c4d96d-3bd3-453f-bdf1-5eb06bd89181 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee54e6f8589a4f69b9cf1428ce5c09a7'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:38.408022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-73c4d96d-3bd3-453f-bdf1-5eb06bd89181 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee54e6f8589a4f69b9cf1428ce5c09a7'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:38.409158 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-73c4d96d-3bd3-453f-bdf1-5eb06bd89181 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee54e6f8589a4f69b9cf1428ce5c09a7'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:38.420244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1265/2531] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:38 2022] DELETE /identity/v3/users/ee54e6f8589a4f69b9cf1428ce5c09a7 => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:38.825418 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc559a3-5540-4161-99cc-0ab453fe6fff None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:38.825948 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc559a3-5540-4161-99cc-0ab453fe6fff None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:38.825948 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc559a3-5540-4161-99cc-0ab453fe6fff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:38.877172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-7fc559a3-5540-4161-99cc-0ab453fe6fff None None] MFA Rules not processed for user `8a6d28b8bb5d46c1940cda041d1c56b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:38.880811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7fc559a3-5540-4161-99cc-0ab453fe6fff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:38.983662 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c4d96d-3bd3-453f-bdf1-5eb06bd89181 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:38.985171 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-73c4d96d-3bd3-453f-bdf1-5eb06bd89181 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:39.014079 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1267/2532] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:38 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:39.034361 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc559a3-5540-4161-99cc-0ab453fe6fff None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:39.039142 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7fc559a3-5540-4161-99cc-0ab453fe6fff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:39.107433 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c4d96d-3bd3-453f-bdf1-5eb06bd89181 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:39.112638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5c8b26-faeb-4694-b1a8-520271ad7e76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:39.113979 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5c8b26-faeb-4694-b1a8-520271ad7e76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:39.114431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5c8b26-faeb-4694-b1a8-520271ad7e76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:39.114868 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5c8b26-faeb-4694-b1a8-520271ad7e76 admin admin] PATH_INFO: `/v3/projects/53bc05af028c40ef83456e69e9668799` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:39.116974 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5c8b26-faeb-4694-b1a8-520271ad7e76 admin admin] RBAC: Authorizing `identity:delete_project(project_id=53bc05af028c40ef83456e69e9668799)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:39.120647 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5c8b26-faeb-4694-b1a8-520271ad7e76 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:39.146670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:39.227535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc559a3-5540-4161-99cc-0ab453fe6fff None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:39.230158 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c5c8b26-faeb-4694-b1a8-520271ad7e76 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '53bc05af028c40ef83456e69e9668799', 'request_id': 'req-6c5c8b26-faeb-4694-b1a8-520271ad7e76'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:39.238903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c5c8b26-faeb-4694-b1a8-520271ad7e76 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '53bc05af028c40ef83456e69e9668799', 'request_id': 'req-6c5c8b26-faeb-4694-b1a8-520271ad7e76'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:39.258946 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c5c8b26-faeb-4694-b1a8-520271ad7e76 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '53bc05af028c40ef83456e69e9668799', 'request_id': 'req-6c5c8b26-faeb-4694-b1a8-520271ad7e76'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:39.286738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c5c8b26-faeb-4694-b1a8-520271ad7e76 admin admin] The token cache is being invalidate because project 53bc05af028c40ef83456e69e9668799 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:39.287231 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c5c8b26-faeb-4694-b1a8-520271ad7e76 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:39.289546 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c5c8b26-faeb-4694-b1a8-520271ad7e76 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:39.300356 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1266/2533] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:38 2022] DELETE /identity/v3/projects/53bc05af028c40ef83456e69e9668799 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:37:39.302021 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048f028e-36f1-4356-a03a-b287c6d75575 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:39.302308 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048f028e-36f1-4356-a03a-b287c6d75575 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:39.303729 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048f028e-36f1-4356-a03a-b287c6d75575 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:39.333534 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-048f028e-36f1-4356-a03a-b287c6d75575 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:39.334176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-048f028e-36f1-4356-a03a-b287c6d75575 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:39.360912 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:39.378964 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff36a9a-4be5-435a-b89b-1cc3b09cf271 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:39.385219 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff36a9a-4be5-435a-b89b-1cc3b09cf271 service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:39.385944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff36a9a-4be5-435a-b89b-1cc3b09cf271 service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:39.387784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff36a9a-4be5-435a-b89b-1cc3b09cf271 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:39.393582 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0ff36a9a-4be5-435a-b89b-1cc3b09cf271 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:39.444219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1267/2534] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:39.444219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048f028e-36f1-4356-a03a-b287c6d75575 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:39.449871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-048f028e-36f1-4356-a03a-b287c6d75575 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:39.477824 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff36a9a-4be5-435a-b89b-1cc3b09cf271 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:39.479439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff36a9a-4be5-435a-b89b-1cc3b09cf271 service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:39.498077 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1268/2535] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:39 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 466 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:39.499735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60788d2-823f-4d64-9906-2ab799a26cd4 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:39.500037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60788d2-823f-4d64-9906-2ab799a26cd4 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:39.500372 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60788d2-823f-4d64-9906-2ab799a26cd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:39.549320 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048f028e-36f1-4356-a03a-b287c6d75575 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:39.554242 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9ab12b-75b1-45e4-a76e-661c54c9c841 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:39.555258 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ab12b-75b1-45e4-a76e-661c54c9c841 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:39.555660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ab12b-75b1-45e4-a76e-661c54c9c841 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:39.556078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ab12b-75b1-45e4-a76e-661c54c9c841 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:39.559578 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9f9ab12b-75b1-45e4-a76e-661c54c9c841 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:39.582885 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9ab12b-75b1-45e4-a76e-661c54c9c841 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:39.584767 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9ab12b-75b1-45e4-a76e-661c54c9c841 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:39.597188 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1268/2536] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:39.613320 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9ab12b-75b1-45e4-a76e-661c54c9c841 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:39.615227 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9f9ab12b-75b1-45e4-a76e-661c54c9c841 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:39.763350 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9ab12b-75b1-45e4-a76e-661c54c9c841 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:39.769707 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff1f34f-4808-48ea-96ee-c01e1a23a1d6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:39.770672 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff1f34f-4808-48ea-96ee-c01e1a23a1d6 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:39.771346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff1f34f-4808-48ea-96ee-c01e1a23a1d6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:39.772513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff1f34f-4808-48ea-96ee-c01e1a23a1d6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:39.776861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-8ff1f34f-4808-48ea-96ee-c01e1a23a1d6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:39.798878 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-a60788d2-823f-4d64-9906-2ab799a26cd4 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:39.800299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a60788d2-823f-4d64-9906-2ab799a26cd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:39.812035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1269/2537] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:39.818797 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60788d2-823f-4d64-9906-2ab799a26cd4 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:39.830842 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60788d2-823f-4d64-9906-2ab799a26cd4 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:39.838770 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b0d3f0-d9d4-4a8b-9fe9-15f0ef52a969 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:39.839104 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b0d3f0-d9d4-4a8b-9fe9-15f0ef52a969 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:39.839479 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b0d3f0-d9d4-4a8b-9fe9-15f0ef52a969 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:39.839986 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b0d3f0-d9d4-4a8b-9fe9-15f0ef52a969 admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:39.840790 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b0d3f0-d9d4-4a8b-9fe9-15f0ef52a969 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:37:39.841192 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b0d3f0-d9d4-4a8b-9fe9-15f0ef52a969 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:39.843015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b0d3f0-d9d4-4a8b-9fe9-15f0ef52a969 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:39.861392 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1270/2538] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:37:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 14:37:39.863453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b0d3f0-d9d4-4a8b-9fe9-15f0ef52a969 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:39.900534 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b0d3f0-d9d4-4a8b-9fe9-15f0ef52a969 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:39.906116 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff1f34f-4808-48ea-96ee-c01e1a23a1d6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:39.907824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff1f34f-4808-48ea-96ee-c01e1a23a1d6 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:39.912169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d75861e-28d8-444c-add3-960119ae6c13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:39.913427 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d75861e-28d8-444c-add3-960119ae6c13 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:39.914660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d75861e-28d8-444c-add3-960119ae6c13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:39.915394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d75861e-28d8-444c-add3-960119ae6c13 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:39.917001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d75861e-28d8-444c-add3-960119ae6c13 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:39.921330 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d75861e-28d8-444c-add3-960119ae6c13 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:39.926274 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1269/2539] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:39 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:40.011169 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1271/2540] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:39 2022] POST /identity/v3/projects => generated 371 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:40.012541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff1f34f-4808-48ea-96ee-c01e1a23a1d6 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:40.032751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff1f34f-4808-48ea-96ee-c01e1a23a1d6 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:40.052627 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:40.126990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f62e152-095b-4099-819d-a151eec81382 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:40.129092 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f62e152-095b-4099-819d-a151eec81382 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:40.129491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f62e152-095b-4099-819d-a151eec81382 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:40.129772 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f62e152-095b-4099-819d-a151eec81382 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:40.130796 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f62e152-095b-4099-819d-a151eec81382 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:40.135729 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f62e152-095b-4099-819d-a151eec81382 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:40.179192 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1270/2541] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:40 2022] POST /identity/v3/users => generated 286 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:40.179374 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d75861e-28d8-444c-add3-960119ae6c13 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:40.196081 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d75861e-28d8-444c-add3-960119ae6c13 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:40.212425 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adea0628-6e51-43f6-bf87-31641b2c9b8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:40.213374 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adea0628-6e51-43f6-bf87-31641b2c9b8e admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:40.213783 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adea0628-6e51-43f6-bf87-31641b2c9b8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:40.214071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adea0628-6e51-43f6-bf87-31641b2c9b8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:40.214729 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adea0628-6e51-43f6-bf87-31641b2c9b8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:40.218878 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adea0628-6e51-43f6-bf87-31641b2c9b8e admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:40.241961 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1272/2542] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:40 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:40.242527 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f62e152-095b-4099-819d-a151eec81382 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:40.259006 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f62e152-095b-4099-819d-a151eec81382 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:40.293870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928a1c8b-ea18-47e6-98f4-b1b33b494d37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:40.294646 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928a1c8b-ea18-47e6-98f4-b1b33b494d37 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:40.295344 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928a1c8b-ea18-47e6-98f4-b1b33b494d37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:40.295344 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928a1c8b-ea18-47e6-98f4-b1b33b494d37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:40.296159 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928a1c8b-ea18-47e6-98f4-b1b33b494d37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:40.298131 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928a1c8b-ea18-47e6-98f4-b1b33b494d37 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:40.315692 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adea0628-6e51-43f6-bf87-31641b2c9b8e admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:40.322679 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1271/2543] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:40 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:40.328057 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adea0628-6e51-43f6-bf87-31641b2c9b8e admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:40.334585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37209a14-9840-46ce-8d09-f2ed3c873499 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:40.335870 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37209a14-9840-46ce-8d09-f2ed3c873499 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:40.336190 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37209a14-9840-46ce-8d09-f2ed3c873499 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:40.336497 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37209a14-9840-46ce-8d09-f2ed3c873499 admin admin] PATH_INFO: `/v3/projects/9a72178bc7cc4b50876fbcec64272e6c/users/111be896666c4f7c9c854009190fc556/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:40.352713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:40.366509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37209a14-9840-46ce-8d09-f2ed3c873499 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a72178bc7cc4b50876fbcec64272e6c, user_id=111be896666c4f7c9c854009190fc556, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:40.373272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37209a14-9840-46ce-8d09-f2ed3c873499 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:40.400336 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1273/2544] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:40 2022] PUT /identity/v3/projects/9a72178bc7cc4b50876fbcec64272e6c/users/111be896666c4f7c9c854009190fc556/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:40.414067 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecc9c26-16bb-47f9-880e-87093c3e9346 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:40.414544 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecc9c26-16bb-47f9-880e-87093c3e9346 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:40.415041 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecc9c26-16bb-47f9-880e-87093c3e9346 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:40.448614 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-1ecc9c26-16bb-47f9-880e-87093c3e9346 None None] MFA Rules not processed for user `111be896666c4f7c9c854009190fc556`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:40.450399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-1ecc9c26-16bb-47f9-880e-87093c3e9346 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:40.552888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1272/2545] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:40 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:40.565290 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37209a14-9840-46ce-8d09-f2ed3c873499 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:40.647932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37209a14-9840-46ce-8d09-f2ed3c873499 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:40.653622 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0fd5bd-5831-4108-92ed-db30efc4ef40 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:40.654625 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0fd5bd-5831-4108-92ed-db30efc4ef40 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:40.655269 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0fd5bd-5831-4108-92ed-db30efc4ef40 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:40.656578 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0fd5bd-5831-4108-92ed-db30efc4ef40 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:40.672789 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0fd5bd-5831-4108-92ed-db30efc4ef40 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:40.674805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0fd5bd-5831-4108-92ed-db30efc4ef40 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:40.688035 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1274/2546] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:40 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:40.713094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecc9c26-16bb-47f9-880e-87093c3e9346 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:40.848025 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecc9c26-16bb-47f9-880e-87093c3e9346 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:40.859332 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119b346f-2242-48fc-85e0-aa576942b9cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:40.859995 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119b346f-2242-48fc-85e0-aa576942b9cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:40.860408 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119b346f-2242-48fc-85e0-aa576942b9cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:40.860558 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119b346f-2242-48fc-85e0-aa576942b9cc admin admin] PATH_INFO: `/v3/users/111be896666c4f7c9c854009190fc556` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:40.865674 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119b346f-2242-48fc-85e0-aa576942b9cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=111be896666c4f7c9c854009190fc556)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:40.867817 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119b346f-2242-48fc-85e0-aa576942b9cc admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:40.921523 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:40.968511 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-119b346f-2242-48fc-85e0-aa576942b9cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '111be896666c4f7c9c854009190fc556'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:40.984446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-119b346f-2242-48fc-85e0-aa576942b9cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '111be896666c4f7c9c854009190fc556'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:40.998502 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-119b346f-2242-48fc-85e0-aa576942b9cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '111be896666c4f7c9c854009190fc556'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:41.039785 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-119b346f-2242-48fc-85e0-aa576942b9cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '111be896666c4f7c9c854009190fc556'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:41.040854 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-119b346f-2242-48fc-85e0-aa576942b9cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '111be896666c4f7c9c854009190fc556'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:41.059349 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1273/2547] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:40 2022] DELETE /identity/v3/users/111be896666c4f7c9c854009190fc556 => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:41.061100 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490a83ba-09c0-4816-be7a-0f3b5148fed5 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:41.061604 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490a83ba-09c0-4816-be7a-0f3b5148fed5 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:41.061816 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490a83ba-09c0-4816-be7a-0f3b5148fed5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:41.084217 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-490a83ba-09c0-4816-be7a-0f3b5148fed5 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:41.086888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-490a83ba-09c0-4816-be7a-0f3b5148fed5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:41.247195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fdac14-f1f7-491e-bc9c-ec79dbce82c4 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:41.247746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fdac14-f1f7-491e-bc9c-ec79dbce82c4 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:41.247746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fdac14-f1f7-491e-bc9c-ec79dbce82c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:41.251618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1275/2548] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:41.279368 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490a83ba-09c0-4816-be7a-0f3b5148fed5 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:41.281176 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-490a83ba-09c0-4816-be7a-0f3b5148fed5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:41.302735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-68fdac14-f1f7-491e-bc9c-ec79dbce82c4 None None] MFA Rules not processed for user `33fb70b5ab4942a38771709eeb3a5789`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:41.302954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-68fdac14-f1f7-491e-bc9c-ec79dbce82c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:41.387706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490a83ba-09c0-4816-be7a-0f3b5148fed5 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:41.391946 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db77241-07c6-4b6f-b324-76d435bce3cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:41.392592 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db77241-07c6-4b6f-b324-76d435bce3cb service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:41.392991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db77241-07c6-4b6f-b324-76d435bce3cb service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:41.393954 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db77241-07c6-4b6f-b324-76d435bce3cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:41.404222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db77241-07c6-4b6f-b324-76d435bce3cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:41.409818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db77241-07c6-4b6f-b324-76d435bce3cb service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:41.411465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1274/2549] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:41 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:37:41.417752 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1276/2550] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:41.809645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fdac14-f1f7-491e-bc9c-ec79dbce82c4 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:41.811478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-68fdac14-f1f7-491e-bc9c-ec79dbce82c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:41.839663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fdac14-f1f7-491e-bc9c-ec79dbce82c4 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:41.852467 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e61a08-817d-41ab-8cfd-57b04655c2a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:41.853541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e61a08-817d-41ab-8cfd-57b04655c2a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:41.854025 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e61a08-817d-41ab-8cfd-57b04655c2a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:41.854412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e61a08-817d-41ab-8cfd-57b04655c2a1 admin admin] PATH_INFO: `/v3/projects/9a72178bc7cc4b50876fbcec64272e6c` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:41.861818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e61a08-817d-41ab-8cfd-57b04655c2a1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a72178bc7cc4b50876fbcec64272e6c)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:41.864507 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e61a08-817d-41ab-8cfd-57b04655c2a1 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:41.941249 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-73e61a08-817d-41ab-8cfd-57b04655c2a1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a72178bc7cc4b50876fbcec64272e6c', 'request_id': 'req-73e61a08-817d-41ab-8cfd-57b04655c2a1'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:41.949526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-73e61a08-817d-41ab-8cfd-57b04655c2a1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a72178bc7cc4b50876fbcec64272e6c', 'request_id': 'req-73e61a08-817d-41ab-8cfd-57b04655c2a1'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:41.956619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-73e61a08-817d-41ab-8cfd-57b04655c2a1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a72178bc7cc4b50876fbcec64272e6c', 'request_id': 'req-73e61a08-817d-41ab-8cfd-57b04655c2a1'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:41.978287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-73e61a08-817d-41ab-8cfd-57b04655c2a1 admin admin] The token cache is being invalidate because project 9a72178bc7cc4b50876fbcec64272e6c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:41.978287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-73e61a08-817d-41ab-8cfd-57b04655c2a1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:41.979224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-73e61a08-817d-41ab-8cfd-57b04655c2a1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:41.986279 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1275/2551] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:41 2022] DELETE /identity/v3/projects/9a72178bc7cc4b50876fbcec64272e6c => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:41.987957 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ff961a-1668-4640-85d2-1a406a2c693b None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:41.988560 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ff961a-1668-4640-85d2-1a406a2c693b None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:41.988963 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ff961a-1668-4640-85d2-1a406a2c693b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:42.017950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-e1ff961a-1668-4640-85d2-1a406a2c693b None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:42.019274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-e1ff961a-1668-4640-85d2-1a406a2c693b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:42.063857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:42.114293 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1277/2552] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:42.123371 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33192773-f003-4d05-86ac-d0d1abb0b79f None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:42.123679 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33192773-f003-4d05-86ac-d0d1abb0b79f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:42.124359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33192773-f003-4d05-86ac-d0d1abb0b79f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:42.173721 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-33192773-f003-4d05-86ac-d0d1abb0b79f None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:42.175431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-33192773-f003-4d05-86ac-d0d1abb0b79f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:42.195369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ff961a-1668-4640-85d2-1a406a2c693b None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:42.196049 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1276/2553] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:42.208502 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ff961a-1668-4640-85d2-1a406a2c693b None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:42.219249 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09e34f4-99e5-43ba-9492-e40d1639f72f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:42.221294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09e34f4-99e5-43ba-9492-e40d1639f72f admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:42.221865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09e34f4-99e5-43ba-9492-e40d1639f72f admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:42.222054 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09e34f4-99e5-43ba-9492-e40d1639f72f admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:42.222771 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09e34f4-99e5-43ba-9492-e40d1639f72f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:37:42.223296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09e34f4-99e5-43ba-9492-e40d1639f72f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:42.227404 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09e34f4-99e5-43ba-9492-e40d1639f72f admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:42.246499 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1278/2554] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:37:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:37:42.250887 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33192773-f003-4d05-86ac-d0d1abb0b79f None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:42.265416 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33192773-f003-4d05-86ac-d0d1abb0b79f None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:42.276406 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c454a4d-e93b-49b3-a0e4-9f80dc6250ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:42.276867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c454a4d-e93b-49b3-a0e4-9f80dc6250ff admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:42.277189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c454a4d-e93b-49b3-a0e4-9f80dc6250ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:42.277466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c454a4d-e93b-49b3-a0e4-9f80dc6250ff admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:42.279064 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c454a4d-e93b-49b3-a0e4-9f80dc6250ff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:42.281518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c454a4d-e93b-49b3-a0e4-9f80dc6250ff admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:42.351963 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1277/2555] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:42 2022] POST /identity/v3/projects => generated 375 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:37:42.355754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09e34f4-99e5-43ba-9492-e40d1639f72f admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:42.368708 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09e34f4-99e5-43ba-9492-e40d1639f72f admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:42.482555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2408e8-a8ad-4234-9390-4dc2af1df8ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:42.483760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2408e8-a8ad-4234-9390-4dc2af1df8ff admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:42.484375 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2408e8-a8ad-4234-9390-4dc2af1df8ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:42.484906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2408e8-a8ad-4234-9390-4dc2af1df8ff admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:42.488235 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2408e8-a8ad-4234-9390-4dc2af1df8ff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:42.492041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2408e8-a8ad-4234-9390-4dc2af1df8ff admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:42.522494 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1279/2556] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:42 2022] POST /identity/v3/users => generated 288 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:42.527140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c454a4d-e93b-49b3-a0e4-9f80dc6250ff admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:42.536162 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:42.542051 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c454a4d-e93b-49b3-a0e4-9f80dc6250ff admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:42.548421 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06825c29-e044-47a7-bb77-fd0318c3b4c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:42.549471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06825c29-e044-47a7-bb77-fd0318c3b4c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:42.550000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06825c29-e044-47a7-bb77-fd0318c3b4c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:42.550482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06825c29-e044-47a7-bb77-fd0318c3b4c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:42.551577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06825c29-e044-47a7-bb77-fd0318c3b4c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:42.554194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06825c29-e044-47a7-bb77-fd0318c3b4c6 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:42.571363 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1278/2557] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:42 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:42.572840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2408e8-a8ad-4234-9390-4dc2af1df8ff admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:42.593112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2408e8-a8ad-4234-9390-4dc2af1df8ff admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:42.603358 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0cff40-547f-4ef9-a9e8-e1e8f0a48659 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:42.603999 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0cff40-547f-4ef9-a9e8-e1e8f0a48659 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:42.604391 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0cff40-547f-4ef9-a9e8-e1e8f0a48659 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:42.604674 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0cff40-547f-4ef9-a9e8-e1e8f0a48659 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:42.605492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0cff40-547f-4ef9-a9e8-e1e8f0a48659 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:42.607272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0cff40-547f-4ef9-a9e8-e1e8f0a48659 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:42.623538 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06825c29-e044-47a7-bb77-fd0318c3b4c6 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:42.632340 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1280/2558] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:42 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:42.638113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06825c29-e044-47a7-bb77-fd0318c3b4c6 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:42.643505 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d017245f-43a7-49aa-bb0a-4472e3c35981 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:42.644189 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d017245f-43a7-49aa-bb0a-4472e3c35981 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:42.644520 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d017245f-43a7-49aa-bb0a-4472e3c35981 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:42.644798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d017245f-43a7-49aa-bb0a-4472e3c35981 admin admin] PATH_INFO: `/v3/projects/17d4f187d45246e68f294e93a0215c7d/users/3b80213b0f5044f692db83f44930fe2b/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:42.669199 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d017245f-43a7-49aa-bb0a-4472e3c35981 admin admin] RBAC: Authorizing `identity:create_grant(project_id=17d4f187d45246e68f294e93a0215c7d, user_id=3b80213b0f5044f692db83f44930fe2b, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:42.672553 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d017245f-43a7-49aa-bb0a-4472e3c35981 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:42.692695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1279/2559] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:42 2022] PUT /identity/v3/projects/17d4f187d45246e68f294e93a0215c7d/users/3b80213b0f5044f692db83f44930fe2b/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:42.705695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513f3f0d-04b0-41e9-941e-b356064bf801 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:42.706098 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513f3f0d-04b0-41e9-941e-b356064bf801 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:42.706487 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513f3f0d-04b0-41e9-941e-b356064bf801 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:42.744445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-513f3f0d-04b0-41e9-941e-b356064bf801 None None] MFA Rules not processed for user `3b80213b0f5044f692db83f44930fe2b`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:42.745567 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-513f3f0d-04b0-41e9-941e-b356064bf801 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:42.820998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1281/2560] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:42 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:42.831979 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d017245f-43a7-49aa-bb0a-4472e3c35981 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:42.833711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d017245f-43a7-49aa-bb0a-4472e3c35981 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:42.909081 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d017245f-43a7-49aa-bb0a-4472e3c35981 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:42.914426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fdc106-3622-401a-b07a-4e7069f062cc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:42.915056 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fdc106-3622-401a-b07a-4e7069f062cc service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:42.915382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fdc106-3622-401a-b07a-4e7069f062cc service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:42.915652 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fdc106-3622-401a-b07a-4e7069f062cc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:42.925746 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fdc106-3622-401a-b07a-4e7069f062cc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:42.927196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fdc106-3622-401a-b07a-4e7069f062cc service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:42.941244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1280/2561] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:42 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:43.717862 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513f3f0d-04b0-41e9-941e-b356064bf801 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:43.719802 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-513f3f0d-04b0-41e9-941e-b356064bf801 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:43.794765 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:43.932049 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513f3f0d-04b0-41e9-941e-b356064bf801 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:43.946221 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52492a1b-571e-46b0-b474-6e7bc3591873 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '436deb3d60c54a799700986519b2ef68', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:43.947098 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52492a1b-571e-46b0-b474-6e7bc3591873 service placement] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:43.947556 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52492a1b-571e-46b0-b474-6e7bc3591873 service placement] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:43.948037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52492a1b-571e-46b0-b474-6e7bc3591873 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:43.950904 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-52492a1b-571e-46b0-b474-6e7bc3591873 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:43.977780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52492a1b-571e-46b0-b474-6e7bc3591873 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:43.979710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52492a1b-571e-46b0-b474-6e7bc3591873 service placement] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:44.010880 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1282/2562] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:43 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:44.558604 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fdc106-3622-401a-b07a-4e7069f062cc service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:44.561167 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-55fdc106-3622-401a-b07a-4e7069f062cc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:44.663187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fdc106-3622-401a-b07a-4e7069f062cc service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:44.668264 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affd4b2e-f4ce-446c-8092-5032fcd97bc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:44.669573 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affd4b2e-f4ce-446c-8092-5032fcd97bc0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:44.669886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affd4b2e-f4ce-446c-8092-5032fcd97bc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:44.670397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affd4b2e-f4ce-446c-8092-5032fcd97bc0 admin admin] PATH_INFO: `/v3/users/54b3856eb61a487dab21d8ffa345701c` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:44.671792 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affd4b2e-f4ce-446c-8092-5032fcd97bc0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=54b3856eb61a487dab21d8ffa345701c)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:44.673938 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affd4b2e-f4ce-446c-8092-5032fcd97bc0 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:44.731015 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-affd4b2e-f4ce-446c-8092-5032fcd97bc0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '54b3856eb61a487dab21d8ffa345701c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:44.749723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-affd4b2e-f4ce-446c-8092-5032fcd97bc0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '54b3856eb61a487dab21d8ffa345701c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:44.773762 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-affd4b2e-f4ce-446c-8092-5032fcd97bc0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '54b3856eb61a487dab21d8ffa345701c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:44.792594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-affd4b2e-f4ce-446c-8092-5032fcd97bc0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '54b3856eb61a487dab21d8ffa345701c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:44.793352 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-affd4b2e-f4ce-446c-8092-5032fcd97bc0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '54b3856eb61a487dab21d8ffa345701c'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:44.795866 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:44.817018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1281/2563] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:44 2022] DELETE /identity/v3/users/54b3856eb61a487dab21d8ffa345701c => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:45.085650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52492a1b-571e-46b0-b474-6e7bc3591873 service placement] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:45.122794 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52492a1b-571e-46b0-b474-6e7bc3591873 service placement] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:45.235658 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f0b928-d32f-4032-a4d6-6d9a268f4f6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:45.236223 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f0b928-d32f-4032-a4d6-6d9a268f4f6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:45.236741 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f0b928-d32f-4032-a4d6-6d9a268f4f6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:45.237084 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f0b928-d32f-4032-a4d6-6d9a268f4f6c admin admin] PATH_INFO: `/v3/projects/1e5731dbee8946a281caefaeb02bd7ff` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:45.238400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f0b928-d32f-4032-a4d6-6d9a268f4f6c admin admin] RBAC: Authorizing `identity:delete_project(project_id=1e5731dbee8946a281caefaeb02bd7ff)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:45.240565 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f0b928-d32f-4032-a4d6-6d9a268f4f6c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:45.320871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-48f0b928-d32f-4032-a4d6-6d9a268f4f6c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1e5731dbee8946a281caefaeb02bd7ff', 'request_id': 'req-48f0b928-d32f-4032-a4d6-6d9a268f4f6c'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:45.327906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-48f0b928-d32f-4032-a4d6-6d9a268f4f6c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1e5731dbee8946a281caefaeb02bd7ff', 'request_id': 'req-48f0b928-d32f-4032-a4d6-6d9a268f4f6c'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:45.341294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-48f0b928-d32f-4032-a4d6-6d9a268f4f6c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1e5731dbee8946a281caefaeb02bd7ff', 'request_id': 'req-48f0b928-d32f-4032-a4d6-6d9a268f4f6c'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:45.356463 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-48f0b928-d32f-4032-a4d6-6d9a268f4f6c admin admin] The token cache is being invalidate because project 1e5731dbee8946a281caefaeb02bd7ff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:45.356817 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-48f0b928-d32f-4032-a4d6-6d9a268f4f6c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:45.357656 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-48f0b928-d32f-4032-a4d6-6d9a268f4f6c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:45.359253 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1283/2564] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:45 2022] DELETE /identity/v3/projects/1e5731dbee8946a281caefaeb02bd7ff => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:45.364928 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2e34a0-c3d6-4110-91c6-0792bb919e0e None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:45.364928 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2e34a0-c3d6-4110-91c6-0792bb919e0e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:45.365391 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2e34a0-c3d6-4110-91c6-0792bb919e0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:45.389044 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-6f2e34a0-c3d6-4110-91c6-0792bb919e0e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:45.390257 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6f2e34a0-c3d6-4110-91c6-0792bb919e0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:45.454397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1282/2565] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:45.462741 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc251fa5-dbd2-435d-a086-c57cab86dc96 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:45.463092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc251fa5-dbd2-435d-a086-c57cab86dc96 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:45.463376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc251fa5-dbd2-435d-a086-c57cab86dc96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:45.484193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-bc251fa5-dbd2-435d-a086-c57cab86dc96 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:45.485513 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-bc251fa5-dbd2-435d-a086-c57cab86dc96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:45.494761 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1284/2566] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:45.499110 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2e34a0-c3d6-4110-91c6-0792bb919e0e None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:45.510630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2e34a0-c3d6-4110-91c6-0792bb919e0e None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:45.515288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbae8d6a-e6fc-48ff-aa22-7904927d5c4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:45.515903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbae8d6a-e6fc-48ff-aa22-7904927d5c4c admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:45.516175 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbae8d6a-e6fc-48ff-aa22-7904927d5c4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:45.516484 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbae8d6a-e6fc-48ff-aa22-7904927d5c4c admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:45.517140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbae8d6a-e6fc-48ff-aa22-7904927d5c4c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:37:45.517440 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbae8d6a-e6fc-48ff-aa22-7904927d5c4c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:45.519186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbae8d6a-e6fc-48ff-aa22-7904927d5c4c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:45.533230 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1283/2567] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:37:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:37:45.537736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc251fa5-dbd2-435d-a086-c57cab86dc96 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:45.547433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc251fa5-dbd2-435d-a086-c57cab86dc96 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:45.552435 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da76f5a-1fa0-4881-a0f7-c6e4491731cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:45.553685 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da76f5a-1fa0-4881-a0f7-c6e4491731cf admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:45.553971 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da76f5a-1fa0-4881-a0f7-c6e4491731cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:45.554419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da76f5a-1fa0-4881-a0f7-c6e4491731cf admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:45.555065 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da76f5a-1fa0-4881-a0f7-c6e4491731cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:45.557034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da76f5a-1fa0-4881-a0f7-c6e4491731cf admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:45.580416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:45.588667 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1285/2568] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:45 2022] POST /identity/v3/projects => generated 363 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:45.592867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbae8d6a-e6fc-48ff-aa22-7904927d5c4c admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:45.603124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbae8d6a-e6fc-48ff-aa22-7904927d5c4c admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:45.668722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946f67ae-8da3-43dc-bd7c-0649a305612d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:45.669326 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946f67ae-8da3-43dc-bd7c-0649a305612d admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:45.669599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946f67ae-8da3-43dc-bd7c-0649a305612d admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:45.669866 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946f67ae-8da3-43dc-bd7c-0649a305612d admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:45.670517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946f67ae-8da3-43dc-bd7c-0649a305612d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:45.672441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946f67ae-8da3-43dc-bd7c-0649a305612d admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:45.694535 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1284/2569] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:45 2022] POST /identity/v3/users => generated 282 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:37:45.698796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da76f5a-1fa0-4881-a0f7-c6e4491731cf admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:45.709249 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da76f5a-1fa0-4881-a0f7-c6e4491731cf admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:45.714791 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3b823f-0bf8-4cd7-b392-bc273d92147a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:45.715424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3b823f-0bf8-4cd7-b392-bc273d92147a admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:45.715707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3b823f-0bf8-4cd7-b392-bc273d92147a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:45.716149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3b823f-0bf8-4cd7-b392-bc273d92147a admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:45.716955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3b823f-0bf8-4cd7-b392-bc273d92147a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:45.718868 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3b823f-0bf8-4cd7-b392-bc273d92147a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:45.730452 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1286/2570] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:45 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:45.734360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946f67ae-8da3-43dc-bd7c-0649a305612d admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:45.745153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946f67ae-8da3-43dc-bd7c-0649a305612d admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:45.750241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4ba8cc-5cf7-4ef7-bbcc-7af4cef83a48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:45.750839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4ba8cc-5cf7-4ef7-bbcc-7af4cef83a48 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:45.751118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4ba8cc-5cf7-4ef7-bbcc-7af4cef83a48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:45.751391 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4ba8cc-5cf7-4ef7-bbcc-7af4cef83a48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:45.752033 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4ba8cc-5cf7-4ef7-bbcc-7af4cef83a48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:45.753666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4ba8cc-5cf7-4ef7-bbcc-7af4cef83a48 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:45.764660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1285/2571] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:45 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:45.768908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3b823f-0bf8-4cd7-b392-bc273d92147a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:45.779376 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3b823f-0bf8-4cd7-b392-bc273d92147a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:45.784225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6617f79c-bc54-455e-804b-346a8d4107c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:45.784936 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6617f79c-bc54-455e-804b-346a8d4107c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:45.785220 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6617f79c-bc54-455e-804b-346a8d4107c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:45.785501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6617f79c-bc54-455e-804b-346a8d4107c3 admin admin] PATH_INFO: `/v3/projects/6aecf637885042be9d5b883af48dd44e/users/d2808553e9b747e0af138fa16c279cd5/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:45.800792 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6617f79c-bc54-455e-804b-346a8d4107c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6aecf637885042be9d5b883af48dd44e, user_id=d2808553e9b747e0af138fa16c279cd5, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:45.804375 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6617f79c-bc54-455e-804b-346a8d4107c3 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:45.817914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1287/2572] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:45 2022] PUT /identity/v3/projects/6aecf637885042be9d5b883af48dd44e/users/d2808553e9b747e0af138fa16c279cd5/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:45.830168 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e37031b-920e-4f6c-b3e5-36de78bf8ad2 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:45.830456 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e37031b-920e-4f6c-b3e5-36de78bf8ad2 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:45.830731 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e37031b-920e-4f6c-b3e5-36de78bf8ad2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:45.863337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-6e37031b-920e-4f6c-b3e5-36de78bf8ad2 None None] MFA Rules not processed for user `d2808553e9b747e0af138fa16c279cd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:45.866222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6e37031b-920e-4f6c-b3e5-36de78bf8ad2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:45.914531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:45.937022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1286/2573] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:45 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:45.942314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6617f79c-bc54-455e-804b-346a8d4107c3 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:46.008687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6617f79c-bc54-455e-804b-346a8d4107c3 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:46.013688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7353faa4-4bdc-4155-84ca-63f480f49463 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:46.014300 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7353faa4-4bdc-4155-84ca-63f480f49463 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:46.014579 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7353faa4-4bdc-4155-84ca-63f480f49463 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:46.014853 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7353faa4-4bdc-4155-84ca-63f480f49463 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:46.015480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7353faa4-4bdc-4155-84ca-63f480f49463 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:46.017398 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7353faa4-4bdc-4155-84ca-63f480f49463 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:46.049866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1288/2574] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:45 2022] POST /identity/v3/projects => generated 365 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:46.051473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e37031b-920e-4f6c-b3e5-36de78bf8ad2 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:46.065665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e37031b-920e-4f6c-b3e5-36de78bf8ad2 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:46.130441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33047db1-cecf-49b4-94b9-d7f7586b04f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:46.131063 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33047db1-cecf-49b4-94b9-d7f7586b04f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:46.131889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33047db1-cecf-49b4-94b9-d7f7586b04f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:46.132161 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33047db1-cecf-49b4-94b9-d7f7586b04f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:46.132819 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33047db1-cecf-49b4-94b9-d7f7586b04f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:46.134723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33047db1-cecf-49b4-94b9-d7f7586b04f8 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:46.158261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1287/2575] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:46 2022] POST /identity/v3/users => generated 289 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:37:46.159501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7353faa4-4bdc-4155-84ca-63f480f49463 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:46.170147 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7353faa4-4bdc-4155-84ca-63f480f49463 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:46.174799 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5ca427-a361-4a5e-b416-8398affb07b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:46.175390 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5ca427-a361-4a5e-b416-8398affb07b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:46.175671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5ca427-a361-4a5e-b416-8398affb07b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:46.175945 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5ca427-a361-4a5e-b416-8398affb07b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:46.176619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5ca427-a361-4a5e-b416-8398affb07b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:46.178825 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5ca427-a361-4a5e-b416-8398affb07b0 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:46.190152 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1289/2576] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:46 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:46.195623 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33047db1-cecf-49b4-94b9-d7f7586b04f8 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:46.209899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33047db1-cecf-49b4-94b9-d7f7586b04f8 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:46.217028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c01f89-adf7-4d64-966c-cf592d22a5f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:46.217696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c01f89-adf7-4d64-966c-cf592d22a5f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:46.217976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c01f89-adf7-4d64-966c-cf592d22a5f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:46.218253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c01f89-adf7-4d64-966c-cf592d22a5f2 admin admin] PATH_INFO: `/v3/projects/61d9fd3545a441fbb926061a8aac511b/users/db4a05e55776436b8601e1780e1e001a/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:46.235495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c01f89-adf7-4d64-966c-cf592d22a5f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=61d9fd3545a441fbb926061a8aac511b, user_id=db4a05e55776436b8601e1780e1e001a, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:46.238194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c01f89-adf7-4d64-966c-cf592d22a5f2 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:46.251815 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1288/2577] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:46 2022] PUT /identity/v3/projects/61d9fd3545a441fbb926061a8aac511b/users/db4a05e55776436b8601e1780e1e001a/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:46.255951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5ca427-a361-4a5e-b416-8398affb07b0 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:46.267181 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5ca427-a361-4a5e-b416-8398affb07b0 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:46.323763 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f808704-c59a-4e2e-81a1-b8da631ca5ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:46.324407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f808704-c59a-4e2e-81a1-b8da631ca5ed admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:46.324691 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f808704-c59a-4e2e-81a1-b8da631ca5ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:46.324966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f808704-c59a-4e2e-81a1-b8da631ca5ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:46.325601 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f808704-c59a-4e2e-81a1-b8da631ca5ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:46.327193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f808704-c59a-4e2e-81a1-b8da631ca5ed admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:46.338719 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1290/2578] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:46 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:46.343422 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c01f89-adf7-4d64-966c-cf592d22a5f2 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:46.354685 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c01f89-adf7-4d64-966c-cf592d22a5f2 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:46.359584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2f3041-b241-4b4f-93a1-cf97cb9331c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:46.360237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f3041-b241-4b4f-93a1-cf97cb9331c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:46.360546 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f3041-b241-4b4f-93a1-cf97cb9331c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:46.360820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f3041-b241-4b4f-93a1-cf97cb9331c9 admin admin] PATH_INFO: `/v3/projects/61d9fd3545a441fbb926061a8aac511b/users/db4a05e55776436b8601e1780e1e001a/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:46.362773 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2f3041-b241-4b4f-93a1-cf97cb9331c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=61d9fd3545a441fbb926061a8aac511b, user_id=db4a05e55776436b8601e1780e1e001a, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:46.365497 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2f3041-b241-4b4f-93a1-cf97cb9331c9 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:46.376410 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1289/2579] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:46 2022] PUT /identity/v3/projects/61d9fd3545a441fbb926061a8aac511b/users/db4a05e55776436b8601e1780e1e001a/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:46.387943 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f85f12-8c46-42d2-8c05-0484cb5edb32 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:46.388233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f85f12-8c46-42d2-8c05-0484cb5edb32 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:46.388550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f85f12-8c46-42d2-8c05-0484cb5edb32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:46.423394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-20f85f12-8c46-42d2-8c05-0484cb5edb32 None None] MFA Rules not processed for user `db4a05e55776436b8601e1780e1e001a`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:46.424653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-20f85f12-8c46-42d2-8c05-0484cb5edb32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:46.457677 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:46.504688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1291/2580] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:46 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:46.516661 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2f3041-b241-4b4f-93a1-cf97cb9331c9 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:46.518436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-2e2f3041-b241-4b4f-93a1-cf97cb9331c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:46.578248 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2f3041-b241-4b4f-93a1-cf97cb9331c9 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:46.581884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a95ff20-6a85-4f81-8b93-433de207770f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:46.582477 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a95ff20-6a85-4f81-8b93-433de207770f service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:46.582752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a95ff20-6a85-4f81-8b93-433de207770f service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:46.583019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a95ff20-6a85-4f81-8b93-433de207770f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:46.593400 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a95ff20-6a85-4f81-8b93-433de207770f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:46.594871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a95ff20-6a85-4f81-8b93-433de207770f service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:46.607086 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1290/2581] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:46 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:46.694112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f85f12-8c46-42d2-8c05-0484cb5edb32 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:46.704439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f85f12-8c46-42d2-8c05-0484cb5edb32 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:46.707966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec873c2a-0f88-418b-87b0-b381a940da48 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:46.708595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec873c2a-0f88-418b-87b0-b381a940da48 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:46.708871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec873c2a-0f88-418b-87b0-b381a940da48 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:46.709291 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec873c2a-0f88-418b-87b0-b381a940da48 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:46.770711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec873c2a-0f88-418b-87b0-b381a940da48 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:46.772856 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec873c2a-0f88-418b-87b0-b381a940da48 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:46.782307 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1292/2582] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:46 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:48.286148 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c52ca10-0ff1-4408-9f81-9d682b3a8d7e None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:48.286872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c52ca10-0ff1-4408-9f81-9d682b3a8d7e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:48.286872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c52ca10-0ff1-4408-9f81-9d682b3a8d7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:48.306654 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-0c52ca10-0ff1-4408-9f81-9d682b3a8d7e None None] MFA Rules not processed for user `c7028d2ccede4b44b567387bb83c4579`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:48.307841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-0c52ca10-0ff1-4408-9f81-9d682b3a8d7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:48.352671 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:48.370821 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1291/2583] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:48 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:37:49.573033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec873c2a-0f88-418b-87b0-b381a940da48 service cinder] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:49.672493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec873c2a-0f88-418b-87b0-b381a940da48 service cinder] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:49.677885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2449f1e9-d8c3-4c9f-9402-b1508f7f259c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:49.678616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2449f1e9-d8c3-4c9f-9402-b1508f7f259c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:49.679109 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2449f1e9-d8c3-4c9f-9402-b1508f7f259c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:49.679454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2449f1e9-d8c3-4c9f-9402-b1508f7f259c admin admin] PATH_INFO: `/v3/users/d2808553e9b747e0af138fa16c279cd5` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:49.680689 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2449f1e9-d8c3-4c9f-9402-b1508f7f259c admin admin] RBAC: Authorizing `identity:delete_user(user_id=d2808553e9b747e0af138fa16c279cd5)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:49.682662 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2449f1e9-d8c3-4c9f-9402-b1508f7f259c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:49.742851 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2449f1e9-d8c3-4c9f-9402-b1508f7f259c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd2808553e9b747e0af138fa16c279cd5'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:49.743616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2449f1e9-d8c3-4c9f-9402-b1508f7f259c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd2808553e9b747e0af138fa16c279cd5'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:49.744271 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2449f1e9-d8c3-4c9f-9402-b1508f7f259c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd2808553e9b747e0af138fa16c279cd5'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:49.755585 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2449f1e9-d8c3-4c9f-9402-b1508f7f259c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd2808553e9b747e0af138fa16c279cd5'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:49.764647 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:49.774726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-2449f1e9-d8c3-4c9f-9402-b1508f7f259c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd2808553e9b747e0af138fa16c279cd5'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:49.796433 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1293/2584] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:49 2022] DELETE /identity/v3/users/d2808553e9b747e0af138fa16c279cd5 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:49.798618 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ead53d-61c5-4a3c-8f47-dc21296f7640 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:49.799093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ead53d-61c5-4a3c-8f47-dc21296f7640 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:49.799378 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ead53d-61c5-4a3c-8f47-dc21296f7640 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:49.823714 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-74ead53d-61c5-4a3c-8f47-dc21296f7640 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:49.825346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-74ead53d-61c5-4a3c-8f47-dc21296f7640 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:49.889647 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1292/2585] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:49.900252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2449f1e9-d8c3-4c9f-9402-b1508f7f259c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:49.901865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-2449f1e9-d8c3-4c9f-9402-b1508f7f259c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:49.987403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2449f1e9-d8c3-4c9f-9402-b1508f7f259c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:49.991362 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6661bd9-d862-4000-ae3c-e55112fa8c01 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:49.991983 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6661bd9-d862-4000-ae3c-e55112fa8c01 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:49.992274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6661bd9-d862-4000-ae3c-e55112fa8c01 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:49.992595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6661bd9-d862-4000-ae3c-e55112fa8c01 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:50.002773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6661bd9-d862-4000-ae3c-e55112fa8c01 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:50.004498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6661bd9-d862-4000-ae3c-e55112fa8c01 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:50.015115 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1294/2586] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:50.387305 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ead53d-61c5-4a3c-8f47-dc21296f7640 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:50.388869 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-74ead53d-61c5-4a3c-8f47-dc21296f7640 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:50.413531 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ead53d-61c5-4a3c-8f47-dc21296f7640 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:50.418503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca2e0f3-9dc6-419e-bc87-e83f24318d93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:50.419346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca2e0f3-9dc6-419e-bc87-e83f24318d93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:50.419625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca2e0f3-9dc6-419e-bc87-e83f24318d93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:50.419899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca2e0f3-9dc6-419e-bc87-e83f24318d93 admin admin] PATH_INFO: `/v3/projects/6aecf637885042be9d5b883af48dd44e` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:50.421357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca2e0f3-9dc6-419e-bc87-e83f24318d93 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6aecf637885042be9d5b883af48dd44e)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:50.423562 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca2e0f3-9dc6-419e-bc87-e83f24318d93 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:50.488678 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bca2e0f3-9dc6-419e-bc87-e83f24318d93 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6aecf637885042be9d5b883af48dd44e', 'request_id': 'req-bca2e0f3-9dc6-419e-bc87-e83f24318d93'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:50.495991 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bca2e0f3-9dc6-419e-bc87-e83f24318d93 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6aecf637885042be9d5b883af48dd44e', 'request_id': 'req-bca2e0f3-9dc6-419e-bc87-e83f24318d93'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:50.501974 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bca2e0f3-9dc6-419e-bc87-e83f24318d93 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6aecf637885042be9d5b883af48dd44e', 'request_id': 'req-bca2e0f3-9dc6-419e-bc87-e83f24318d93'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:50.514865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bca2e0f3-9dc6-419e-bc87-e83f24318d93 admin admin] The token cache is being invalidate because project 6aecf637885042be9d5b883af48dd44e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:50.515157 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bca2e0f3-9dc6-419e-bc87-e83f24318d93 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:50.515879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bca2e0f3-9dc6-419e-bc87-e83f24318d93 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:50.517363 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1293/2587] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:50 2022] DELETE /identity/v3/projects/6aecf637885042be9d5b883af48dd44e => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:50.521730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6661bd9-d862-4000-ae3c-e55112fa8c01 service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:50.541469 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6661bd9-d862-4000-ae3c-e55112fa8c01 service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:50.605401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4889d3-460f-4d94-8e59-0f930727f3bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:50.606063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4889d3-460f-4d94-8e59-0f930727f3bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:50.606312 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4889d3-460f-4d94-8e59-0f930727f3bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:50.606587 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4889d3-460f-4d94-8e59-0f930727f3bb admin admin] PATH_INFO: `/v3/users/db4a05e55776436b8601e1780e1e001a` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:50.607713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4889d3-460f-4d94-8e59-0f930727f3bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=db4a05e55776436b8601e1780e1e001a)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:50.609713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4889d3-460f-4d94-8e59-0f930727f3bb admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:50.676478 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8b4889d3-460f-4d94-8e59-0f930727f3bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'db4a05e55776436b8601e1780e1e001a'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:50.677155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8b4889d3-460f-4d94-8e59-0f930727f3bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'db4a05e55776436b8601e1780e1e001a'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:50.681618 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8b4889d3-460f-4d94-8e59-0f930727f3bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'db4a05e55776436b8601e1780e1e001a'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:50.707669 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8b4889d3-460f-4d94-8e59-0f930727f3bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'db4a05e55776436b8601e1780e1e001a'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:50.729244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8b4889d3-460f-4d94-8e59-0f930727f3bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'db4a05e55776436b8601e1780e1e001a'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:50.744537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:50.752101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1295/2588] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:50 2022] DELETE /identity/v3/users/db4a05e55776436b8601e1780e1e001a => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:51.122069 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca2e0f3-9dc6-419e-bc87-e83f24318d93 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:51.124471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-bca2e0f3-9dc6-419e-bc87-e83f24318d93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:51.152048 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca2e0f3-9dc6-419e-bc87-e83f24318d93 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:51.227520 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbf07e4-0c9f-48df-962c-a99db7475670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:51.228277 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbf07e4-0c9f-48df-962c-a99db7475670 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:51.228639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbf07e4-0c9f-48df-962c-a99db7475670 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:51.228977 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbf07e4-0c9f-48df-962c-a99db7475670 admin admin] PATH_INFO: `/v3/projects/61d9fd3545a441fbb926061a8aac511b` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:51.230196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbf07e4-0c9f-48df-962c-a99db7475670 admin admin] RBAC: Authorizing `identity:delete_project(project_id=61d9fd3545a441fbb926061a8aac511b)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:51.232400 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbf07e4-0c9f-48df-962c-a99db7475670 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:51.249700 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:51.305473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9dbf07e4-0c9f-48df-962c-a99db7475670 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '61d9fd3545a441fbb926061a8aac511b', 'request_id': 'req-9dbf07e4-0c9f-48df-962c-a99db7475670'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:51.313479 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9dbf07e4-0c9f-48df-962c-a99db7475670 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '61d9fd3545a441fbb926061a8aac511b', 'request_id': 'req-9dbf07e4-0c9f-48df-962c-a99db7475670'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:51.320803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9dbf07e4-0c9f-48df-962c-a99db7475670 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '61d9fd3545a441fbb926061a8aac511b', 'request_id': 'req-9dbf07e4-0c9f-48df-962c-a99db7475670'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:51.340952 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9dbf07e4-0c9f-48df-962c-a99db7475670 admin admin] The token cache is being invalidate because project 61d9fd3545a441fbb926061a8aac511b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:51.341271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9dbf07e4-0c9f-48df-962c-a99db7475670 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:51.342153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9dbf07e4-0c9f-48df-962c-a99db7475670 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:51.343937 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1294/2589] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:51 2022] DELETE /identity/v3/projects/61d9fd3545a441fbb926061a8aac511b => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:51.348688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e56e61-a1fd-4389-8242-afca7a7cd849 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:51.349548 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e56e61-a1fd-4389-8242-afca7a7cd849 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:51.349837 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e56e61-a1fd-4389-8242-afca7a7cd849 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:51.375493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-10e56e61-a1fd-4389-8242-afca7a7cd849 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:51.376766 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-10e56e61-a1fd-4389-8242-afca7a7cd849 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:51.464227 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1296/2590] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:51.472778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a64efd-4315-4cc5-b640-0c9f9f8278cc None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:51.473118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a64efd-4315-4cc5-b640-0c9f9f8278cc None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:51.473441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a64efd-4315-4cc5-b640-0c9f9f8278cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:51.525905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-43a64efd-4315-4cc5-b640-0c9f9f8278cc None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:51.527413 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-43a64efd-4315-4cc5-b640-0c9f9f8278cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:51.554975 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1295/2591] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:37:51.559705 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e56e61-a1fd-4389-8242-afca7a7cd849 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:51.577448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e56e61-a1fd-4389-8242-afca7a7cd849 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:51.584475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822e6aad-2d1b-4da8-8e7a-2d9ccb778e2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:51.585367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822e6aad-2d1b-4da8-8e7a-2d9ccb778e2a admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:51.585809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822e6aad-2d1b-4da8-8e7a-2d9ccb778e2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:51.586332 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822e6aad-2d1b-4da8-8e7a-2d9ccb778e2a admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:51.587154 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822e6aad-2d1b-4da8-8e7a-2d9ccb778e2a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:37:51.588513 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822e6aad-2d1b-4da8-8e7a-2d9ccb778e2a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:51.594621 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822e6aad-2d1b-4da8-8e7a-2d9ccb778e2a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:51.619724 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1297/2592] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:37:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:37:51.624430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a64efd-4315-4cc5-b640-0c9f9f8278cc None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:51.634807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a64efd-4315-4cc5-b640-0c9f9f8278cc None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:51.658973 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdf20d9-fd35-424d-98d5-d33f59eb8528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:51.659679 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdf20d9-fd35-424d-98d5-d33f59eb8528 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:51.660028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdf20d9-fd35-424d-98d5-d33f59eb8528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:51.661258 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdf20d9-fd35-424d-98d5-d33f59eb8528 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:51.662010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdf20d9-fd35-424d-98d5-d33f59eb8528 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:51.667311 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdf20d9-fd35-424d-98d5-d33f59eb8528 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:51.737626 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1296/2593] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:51 2022] POST /identity/v3/projects => generated 379 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:51.743838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822e6aad-2d1b-4da8-8e7a-2d9ccb778e2a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:51.760787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822e6aad-2d1b-4da8-8e7a-2d9ccb778e2a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:51.866402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62be2b32-50d5-47a1-9d35-fbd247f9612b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:51.867615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62be2b32-50d5-47a1-9d35-fbd247f9612b admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:51.867972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62be2b32-50d5-47a1-9d35-fbd247f9612b admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:51.868337 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62be2b32-50d5-47a1-9d35-fbd247f9612b admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:51.869061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62be2b32-50d5-47a1-9d35-fbd247f9612b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:51.871097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62be2b32-50d5-47a1-9d35-fbd247f9612b admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:51.898874 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1298/2594] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:51 2022] POST /identity/v3/users => generated 290 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:37:51.903568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdf20d9-fd35-424d-98d5-d33f59eb8528 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:51.916597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdf20d9-fd35-424d-98d5-d33f59eb8528 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:51.922535 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718b4328-8a1e-48c0-b806-73f36a11d572 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:51.923154 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718b4328-8a1e-48c0-b806-73f36a11d572 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:51.923445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718b4328-8a1e-48c0-b806-73f36a11d572 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:51.923740 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718b4328-8a1e-48c0-b806-73f36a11d572 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:51.924433 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718b4328-8a1e-48c0-b806-73f36a11d572 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:51.926080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718b4328-8a1e-48c0-b806-73f36a11d572 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:51.944723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1297/2595] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:51 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:51.949488 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62be2b32-50d5-47a1-9d35-fbd247f9612b admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:51.962159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62be2b32-50d5-47a1-9d35-fbd247f9612b admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:51.967306 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d042459-567c-4434-9ef4-bf1b46f4e113 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:51.967937 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d042459-567c-4434-9ef4-bf1b46f4e113 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:51.968222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d042459-567c-4434-9ef4-bf1b46f4e113 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:51.968536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d042459-567c-4434-9ef4-bf1b46f4e113 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:51.969204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d042459-567c-4434-9ef4-bf1b46f4e113 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:51.970811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d042459-567c-4434-9ef4-bf1b46f4e113 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:51.984779 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1299/2596] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:51 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:37:51.988594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718b4328-8a1e-48c0-b806-73f36a11d572 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:52.001801 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718b4328-8a1e-48c0-b806-73f36a11d572 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:52.010912 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8f95a8-ebce-44e5-a824-d2f4f707b6b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:52.011703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8f95a8-ebce-44e5-a824-d2f4f707b6b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:52.012106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8f95a8-ebce-44e5-a824-d2f4f707b6b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:52.012522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8f95a8-ebce-44e5-a824-d2f4f707b6b1 admin admin] PATH_INFO: `/v3/projects/37f60715f0a04972a078f3aa96550e4f/users/bf968eccacd142a8be75b655b2cef6e2/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:52.033884 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8f95a8-ebce-44e5-a824-d2f4f707b6b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=37f60715f0a04972a078f3aa96550e4f, user_id=bf968eccacd142a8be75b655b2cef6e2, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:52.036717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8f95a8-ebce-44e5-a824-d2f4f707b6b1 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:52.058030 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1298/2597] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:51 2022] PUT /identity/v3/projects/37f60715f0a04972a078f3aa96550e4f/users/bf968eccacd142a8be75b655b2cef6e2/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:52.070958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a7935b-8708-476f-8e6c-1116f220b8fc None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:52.072740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a7935b-8708-476f-8e6c-1116f220b8fc None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:52.073238 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a7935b-8708-476f-8e6c-1116f220b8fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:52.110168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-33a7935b-8708-476f-8e6c-1116f220b8fc None None] MFA Rules not processed for user `bf968eccacd142a8be75b655b2cef6e2`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:52.111901 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-33a7935b-8708-476f-8e6c-1116f220b8fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:52.186580 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:52.190439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1300/2598] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:52 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:52.195830 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8f95a8-ebce-44e5-a824-d2f4f707b6b1 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:52.395471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8f95a8-ebce-44e5-a824-d2f4f707b6b1 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:52.402872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d00ab04-55fb-4cf9-9710-88304115774a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:52.402872 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d00ab04-55fb-4cf9-9710-88304115774a admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:52.403227 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d00ab04-55fb-4cf9-9710-88304115774a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:52.403527 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d00ab04-55fb-4cf9-9710-88304115774a admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:52.404093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d00ab04-55fb-4cf9-9710-88304115774a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:52.406075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d00ab04-55fb-4cf9-9710-88304115774a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:52.436340 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1299/2599] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:52 2022] POST /identity/v3/projects => generated 381 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:52.438677 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a7935b-8708-476f-8e6c-1116f220b8fc None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:52.455863 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a7935b-8708-476f-8e6c-1116f220b8fc None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:52.515664 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f19321-88a6-41b3-9c3f-9994d0d33710 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:52.516280 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f19321-88a6-41b3-9c3f-9994d0d33710 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:52.516594 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f19321-88a6-41b3-9c3f-9994d0d33710 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:52.516901 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f19321-88a6-41b3-9c3f-9994d0d33710 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:52.517536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f19321-88a6-41b3-9c3f-9994d0d33710 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:52.519464 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f19321-88a6-41b3-9c3f-9994d0d33710 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:52.546308 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1301/2600] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:52 2022] POST /identity/v3/users => generated 297 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:37:52.550850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d00ab04-55fb-4cf9-9710-88304115774a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:52.564025 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d00ab04-55fb-4cf9-9710-88304115774a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:52.571160 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9723f9c-e472-46af-a6e8-876d612e7632 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:52.572086 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9723f9c-e472-46af-a6e8-876d612e7632 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:52.573382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9723f9c-e472-46af-a6e8-876d612e7632 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:52.573382 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9723f9c-e472-46af-a6e8-876d612e7632 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:52.574206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9723f9c-e472-46af-a6e8-876d612e7632 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:52.575982 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9723f9c-e472-46af-a6e8-876d612e7632 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:52.590500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1300/2601] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:52 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:52.594989 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f19321-88a6-41b3-9c3f-9994d0d33710 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:52.607753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f19321-88a6-41b3-9c3f-9994d0d33710 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:52.614059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d38d10-bb8a-4fbe-98e9-d826e63b75cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:52.614777 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d38d10-bb8a-4fbe-98e9-d826e63b75cd admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:52.615061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d38d10-bb8a-4fbe-98e9-d826e63b75cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:52.615495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d38d10-bb8a-4fbe-98e9-d826e63b75cd admin admin] PATH_INFO: `/v3/projects/fa19733707044f81ad9abc87e29b9df3/users/79f6433e80ef47e3aa79065aee6349ef/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:52.651061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d38d10-bb8a-4fbe-98e9-d826e63b75cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa19733707044f81ad9abc87e29b9df3, user_id=79f6433e80ef47e3aa79065aee6349ef, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:52.654675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d38d10-bb8a-4fbe-98e9-d826e63b75cd admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:52.674440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1302/2602] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:52 2022] PUT /identity/v3/projects/fa19733707044f81ad9abc87e29b9df3/users/79f6433e80ef47e3aa79065aee6349ef/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:52.679301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9723f9c-e472-46af-a6e8-876d612e7632 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:52.692428 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9723f9c-e472-46af-a6e8-876d612e7632 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:52.701058 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:52.784252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd72e66-4881-46ba-9c2a-38a3c629dcfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:52.785194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd72e66-4881-46ba-9c2a-38a3c629dcfd admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:52.785472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd72e66-4881-46ba-9c2a-38a3c629dcfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:52.785747 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd72e66-4881-46ba-9c2a-38a3c629dcfd admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:52.786390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd72e66-4881-46ba-9c2a-38a3c629dcfd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:52.788639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd72e66-4881-46ba-9c2a-38a3c629dcfd admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:52.800103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1301/2603] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:52 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:37:52.804435 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d38d10-bb8a-4fbe-98e9-d826e63b75cd admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:52.823865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d38d10-bb8a-4fbe-98e9-d826e63b75cd admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:52.829095 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c808a6b6-d441-433c-853a-c63b5e458ef9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:52.829840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c808a6b6-d441-433c-853a-c63b5e458ef9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:52.830207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c808a6b6-d441-433c-853a-c63b5e458ef9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:52.830377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c808a6b6-d441-433c-853a-c63b5e458ef9 admin admin] PATH_INFO: `/v3/projects/fa19733707044f81ad9abc87e29b9df3/users/79f6433e80ef47e3aa79065aee6349ef/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:52.832580 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c808a6b6-d441-433c-853a-c63b5e458ef9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa19733707044f81ad9abc87e29b9df3, user_id=79f6433e80ef47e3aa79065aee6349ef, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:52.836545 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c808a6b6-d441-433c-853a-c63b5e458ef9 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:52.878924 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1303/2604] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:52 2022] PUT /identity/v3/projects/fa19733707044f81ad9abc87e29b9df3/users/79f6433e80ef47e3aa79065aee6349ef/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:52.910207 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130211d9-8223-49df-bff4-a42a6551c622 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:52.910569 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130211d9-8223-49df-bff4-a42a6551c622 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:52.910916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130211d9-8223-49df-bff4-a42a6551c622 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:52.941725 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-130211d9-8223-49df-bff4-a42a6551c622 None None] MFA Rules not processed for user `79f6433e80ef47e3aa79065aee6349ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:52.942955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-130211d9-8223-49df-bff4-a42a6551c622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:53.014885 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1302/2605] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:52 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:53.028602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c808a6b6-d441-433c-853a-c63b5e458ef9 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:53.031025 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c808a6b6-d441-433c-853a-c63b5e458ef9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:53.095313 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c808a6b6-d441-433c-853a-c63b5e458ef9 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:53.098987 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e2bc37-332a-4d10-b818-1da12f496944 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:53.099587 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e2bc37-332a-4d10-b818-1da12f496944 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:53.099872 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e2bc37-332a-4d10-b818-1da12f496944 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:53.100750 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e2bc37-332a-4d10-b818-1da12f496944 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:53.110210 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e2bc37-332a-4d10-b818-1da12f496944 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:53.111689 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e2bc37-332a-4d10-b818-1da12f496944 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:53.122805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1304/2606] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:53 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:53.479267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130211d9-8223-49df-bff4-a42a6551c622 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:53.488593 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130211d9-8223-49df-bff4-a42a6551c622 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:53.494448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a32139b-1a8f-45a7-b0da-190077caa3d1 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79f6433e80ef47e3aa79065aee6349ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa19733707044f81ad9abc87e29b9df3', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:53.495156 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a32139b-1a8f-45a7-b0da-190077caa3d1 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:53.495502 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a32139b-1a8f-45a7-b0da-190077caa3d1 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:53.495844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a32139b-1a8f-45a7-b0da-190077caa3d1 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:53.496623 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a32139b-1a8f-45a7-b0da-190077caa3d1 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:37:53.496993 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a32139b-1a8f-45a7-b0da-190077caa3d1 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:53.498646 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a32139b-1a8f-45a7-b0da-190077caa3d1 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:53.511167 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1303/2607] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:37:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:37:53.515974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e2bc37-332a-4d10-b818-1da12f496944 service cinder] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:53.525214 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e2bc37-332a-4d10-b818-1da12f496944 service cinder] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:53.531103 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0016784d-7390-4f47-ab02-afe3c1a9a5c6 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79f6433e80ef47e3aa79065aee6349ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa19733707044f81ad9abc87e29b9df3', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:53.531367 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0016784d-7390-4f47-ab02-afe3c1a9a5c6 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:53.531649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0016784d-7390-4f47-ab02-afe3c1a9a5c6 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:53.531919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0016784d-7390-4f47-ab02-afe3c1a9a5c6 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:53.532575 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0016784d-7390-4f47-ab02-afe3c1a9a5c6 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:53.534448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0016784d-7390-4f47-ab02-afe3c1a9a5c6 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:53.563857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1305/2608] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:53 2022] POST /identity/v3/projects => generated 355 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:53.568370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a32139b-1a8f-45a7-b0da-190077caa3d1 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:53.580211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a32139b-1a8f-45a7-b0da-190077caa3d1 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:53.651118 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935296bb-0d72-4526-abb6-17f421c45551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79f6433e80ef47e3aa79065aee6349ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa19733707044f81ad9abc87e29b9df3', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:53.651639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935296bb-0d72-4526-abb6-17f421c45551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:53.651922 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935296bb-0d72-4526-abb6-17f421c45551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:53.652201 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935296bb-0d72-4526-abb6-17f421c45551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:53.653160 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935296bb-0d72-4526-abb6-17f421c45551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:37:53.653466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935296bb-0d72-4526-abb6-17f421c45551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:53.655209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935296bb-0d72-4526-abb6-17f421c45551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:53.674618 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1304/2609] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:37:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:37:53.716242 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0016784d-7390-4f47-ab02-afe3c1a9a5c6 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:53.729294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0016784d-7390-4f47-ab02-afe3c1a9a5c6 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:53.736260 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbd8b52-9256-4fe1-a778-0461ada2b551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79f6433e80ef47e3aa79065aee6349ef', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fa19733707044f81ad9abc87e29b9df3', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:53.737158 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbd8b52-9256-4fe1-a778-0461ada2b551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:53.737671 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbd8b52-9256-4fe1-a778-0461ada2b551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:53.737987 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbd8b52-9256-4fe1-a778-0461ada2b551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] PATH_INFO: `/v3/projects/478e1df8137b4804934db9ea7a9587e5` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:53.739705 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbd8b52-9256-4fe1-a778-0461ada2b551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] RBAC: Authorizing `identity:delete_project(project_id=478e1df8137b4804934db9ea7a9587e5)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:53.743397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbd8b52-9256-4fe1-a778-0461ada2b551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:53.775319 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=93,keystone:INSERT=6,keystone:DELETE=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:53.801797 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-fcbd8b52-9256-4fe1-a778-0461ada2b551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '478e1df8137b4804934db9ea7a9587e5', 'request_id': 'req-fcbd8b52-9256-4fe1-a778-0461ada2b551'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:53.809279 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-fcbd8b52-9256-4fe1-a778-0461ada2b551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '478e1df8137b4804934db9ea7a9587e5', 'request_id': 'req-fcbd8b52-9256-4fe1-a778-0461ada2b551'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:53.815220 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-fcbd8b52-9256-4fe1-a778-0461ada2b551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '478e1df8137b4804934db9ea7a9587e5', 'request_id': 'req-fcbd8b52-9256-4fe1-a778-0461ada2b551'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:53.829174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-fcbd8b52-9256-4fe1-a778-0461ada2b551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] The token cache is being invalidate because project 478e1df8137b4804934db9ea7a9587e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:53.829470 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-fcbd8b52-9256-4fe1-a778-0461ada2b551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:53.830182 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-fcbd8b52-9256-4fe1-a778-0461ada2b551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:53.831940 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1306/2610] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:53 2022] DELETE /identity/v3/projects/478e1df8137b4804934db9ea7a9587e5 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:54.114279 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935296bb-0d72-4526-abb6-17f421c45551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:54.116831 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-935296bb-0d72-4526-abb6-17f421c45551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:54.133265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=100 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:54.189206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935296bb-0d72-4526-abb6-17f421c45551 tempest-VolumeQuotaClassesTest-1387037414 tempest-VolumeQuotaClassesTest-1387037414-project-admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:54.250954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cae7277-7486-4d2f-9957-64f40db646db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:54.251573 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cae7277-7486-4d2f-9957-64f40db646db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:54.251852 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cae7277-7486-4d2f-9957-64f40db646db admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:54.252131 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cae7277-7486-4d2f-9957-64f40db646db admin admin] PATH_INFO: `/v3/users/bf968eccacd142a8be75b655b2cef6e2` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:54.253467 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cae7277-7486-4d2f-9957-64f40db646db admin admin] RBAC: Authorizing `identity:delete_user(user_id=bf968eccacd142a8be75b655b2cef6e2)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:54.255391 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cae7277-7486-4d2f-9957-64f40db646db admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:54.305399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8cae7277-7486-4d2f-9957-64f40db646db admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bf968eccacd142a8be75b655b2cef6e2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:54.314163 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8cae7277-7486-4d2f-9957-64f40db646db admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bf968eccacd142a8be75b655b2cef6e2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:54.322772 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8cae7277-7486-4d2f-9957-64f40db646db admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bf968eccacd142a8be75b655b2cef6e2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:54.337848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8cae7277-7486-4d2f-9957-64f40db646db admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bf968eccacd142a8be75b655b2cef6e2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:54.338650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8cae7277-7486-4d2f-9957-64f40db646db admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bf968eccacd142a8be75b655b2cef6e2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:54.344734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1305/2611] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:54 2022] DELETE /identity/v3/users/bf968eccacd142a8be75b655b2cef6e2 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:54.349531 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d923007d-0854-466f-9255-4ebfc1eb2112 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:54.349841 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d923007d-0854-466f-9255-4ebfc1eb2112 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:54.350669 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d923007d-0854-466f-9255-4ebfc1eb2112 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:54.371850 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-d923007d-0854-466f-9255-4ebfc1eb2112 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:54.373115 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d923007d-0854-466f-9255-4ebfc1eb2112 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:54.454888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1307/2612] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:54.467063 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cae7277-7486-4d2f-9957-64f40db646db admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:54.468656 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-8cae7277-7486-4d2f-9957-64f40db646db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:54.524726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:54.543360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cae7277-7486-4d2f-9957-64f40db646db admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:54.547265 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fa314b-0faf-4b25-82f5-9867fef5132e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:54.547949 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fa314b-0faf-4b25-82f5-9867fef5132e service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:54.548297 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fa314b-0faf-4b25-82f5-9867fef5132e service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:54.548591 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fa314b-0faf-4b25-82f5-9867fef5132e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:54.559515 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fa314b-0faf-4b25-82f5-9867fef5132e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:54.561104 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fa314b-0faf-4b25-82f5-9867fef5132e service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:54.572728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1306/2613] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:54.872541 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d923007d-0854-466f-9255-4ebfc1eb2112 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:54.874108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d923007d-0854-466f-9255-4ebfc1eb2112 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:54.886912 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d923007d-0854-466f-9255-4ebfc1eb2112 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:54.892162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:54.892844 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:54.893128 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:54.893405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76 admin admin] PATH_INFO: `/v3/projects/37f60715f0a04972a078f3aa96550e4f` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:54.894542 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76 admin admin] RBAC: Authorizing `identity:delete_project(project_id=37f60715f0a04972a078f3aa96550e4f)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:54.896488 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:54.944486 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '37f60715f0a04972a078f3aa96550e4f', 'request_id': 'req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:54.950920 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '37f60715f0a04972a078f3aa96550e4f', 'request_id': 'req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:54.957151 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '37f60715f0a04972a078f3aa96550e4f', 'request_id': 'req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:54.971203 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76 admin admin] The token cache is being invalidate because project 37f60715f0a04972a078f3aa96550e4f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:54.971480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:54.972112 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:54.973634 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1308/2614] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:54 2022] DELETE /identity/v3/projects/37f60715f0a04972a078f3aa96550e4f => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:54.977616 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fa314b-0faf-4b25-82f5-9867fef5132e service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:54.987502 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fa314b-0faf-4b25-82f5-9867fef5132e service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:55.050023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06814a3-81a9-4062-9cc2-92022e98e71c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:55.050664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06814a3-81a9-4062-9cc2-92022e98e71c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:55.051029 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06814a3-81a9-4062-9cc2-92022e98e71c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:55.051343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06814a3-81a9-4062-9cc2-92022e98e71c admin admin] PATH_INFO: `/v3/users/79f6433e80ef47e3aa79065aee6349ef` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:55.052674 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06814a3-81a9-4062-9cc2-92022e98e71c admin admin] RBAC: Authorizing `identity:delete_user(user_id=79f6433e80ef47e3aa79065aee6349ef)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:55.055298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06814a3-81a9-4062-9cc2-92022e98e71c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:55.104053 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c06814a3-81a9-4062-9cc2-92022e98e71c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '79f6433e80ef47e3aa79065aee6349ef'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:55.110767 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c06814a3-81a9-4062-9cc2-92022e98e71c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '79f6433e80ef47e3aa79065aee6349ef'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:55.118333 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c06814a3-81a9-4062-9cc2-92022e98e71c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '79f6433e80ef47e3aa79065aee6349ef'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:55.130946 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c06814a3-81a9-4062-9cc2-92022e98e71c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '79f6433e80ef47e3aa79065aee6349ef'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:55.132222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c06814a3-81a9-4062-9cc2-92022e98e71c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '79f6433e80ef47e3aa79065aee6349ef'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:55.138867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1307/2615] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:54 2022] DELETE /identity/v3/users/79f6433e80ef47e3aa79065aee6349ef => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:55.163691 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:55.165504 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:55.181559 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25787d8-8e9c-4a6a-85e7-788b3ebcfd76 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:55.220647 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:55.254999 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30083ac3-8cb2-4095-949a-a9b828d07735 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:55.255922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30083ac3-8cb2-4095-949a-a9b828d07735 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:55.256838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30083ac3-8cb2-4095-949a-a9b828d07735 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:55.257225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30083ac3-8cb2-4095-949a-a9b828d07735 admin admin] PATH_INFO: `/v3/users/3b80213b0f5044f692db83f44930fe2b` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:55.258661 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30083ac3-8cb2-4095-949a-a9b828d07735 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3b80213b0f5044f692db83f44930fe2b)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:55.260703 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30083ac3-8cb2-4095-949a-a9b828d07735 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:55.316077 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-30083ac3-8cb2-4095-949a-a9b828d07735 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3b80213b0f5044f692db83f44930fe2b'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:55.316764 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-30083ac3-8cb2-4095-949a-a9b828d07735 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3b80213b0f5044f692db83f44930fe2b'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:55.317379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-30083ac3-8cb2-4095-949a-a9b828d07735 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3b80213b0f5044f692db83f44930fe2b'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:55.326555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-30083ac3-8cb2-4095-949a-a9b828d07735 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3b80213b0f5044f692db83f44930fe2b'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:55.346571 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-30083ac3-8cb2-4095-949a-a9b828d07735 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3b80213b0f5044f692db83f44930fe2b'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:55.362138 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1309/2616] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:55 2022] DELETE /identity/v3/users/3b80213b0f5044f692db83f44930fe2b => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:55.367366 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab59f5d-86c4-4f53-ad3d-ff28a41e46c2 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:55.367647 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab59f5d-86c4-4f53-ad3d-ff28a41e46c2 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:55.367918 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab59f5d-86c4-4f53-ad3d-ff28a41e46c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:55.408721 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-dab59f5d-86c4-4f53-ad3d-ff28a41e46c2 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:55.409944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-dab59f5d-86c4-4f53-ad3d-ff28a41e46c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:55.485755 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1308/2617] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:55.494038 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30083ac3-8cb2-4095-949a-a9b828d07735 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:55.496200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-30083ac3-8cb2-4095-949a-a9b828d07735 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:55.524360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab59f5d-86c4-4f53-ad3d-ff28a41e46c2 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:55.525777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-dab59f5d-86c4-4f53-ad3d-ff28a41e46c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:55.534517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:55.555954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab59f5d-86c4-4f53-ad3d-ff28a41e46c2 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:55.562593 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bffacb-22f5-4a90-9ccc-359044a437c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:55.563425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bffacb-22f5-4a90-9ccc-359044a437c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:55.564674 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bffacb-22f5-4a90-9ccc-359044a437c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:55.564933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bffacb-22f5-4a90-9ccc-359044a437c3 admin admin] PATH_INFO: `/v3/projects/fa19733707044f81ad9abc87e29b9df3` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:55.566335 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bffacb-22f5-4a90-9ccc-359044a437c3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa19733707044f81ad9abc87e29b9df3)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:55.569037 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bffacb-22f5-4a90-9ccc-359044a437c3 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:55.573653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30083ac3-8cb2-4095-949a-a9b828d07735 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:55.577421 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a69ec16-ffd3-42e0-89d5-b51ab3503c75 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:55.578037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a69ec16-ffd3-42e0-89d5-b51ab3503c75 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:55.578315 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a69ec16-ffd3-42e0-89d5-b51ab3503c75 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:55.578582 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a69ec16-ffd3-42e0-89d5-b51ab3503c75 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:55.588759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a69ec16-ffd3-42e0-89d5-b51ab3503c75 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:55.590230 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a69ec16-ffd3-42e0-89d5-b51ab3503c75 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:55.602093 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1310/2618] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:55.622127 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-12bffacb-22f5-4a90-9ccc-359044a437c3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa19733707044f81ad9abc87e29b9df3', 'request_id': 'req-12bffacb-22f5-4a90-9ccc-359044a437c3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:55.628967 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-12bffacb-22f5-4a90-9ccc-359044a437c3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa19733707044f81ad9abc87e29b9df3', 'request_id': 'req-12bffacb-22f5-4a90-9ccc-359044a437c3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:55.635922 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-12bffacb-22f5-4a90-9ccc-359044a437c3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa19733707044f81ad9abc87e29b9df3', 'request_id': 'req-12bffacb-22f5-4a90-9ccc-359044a437c3'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:55.651014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-12bffacb-22f5-4a90-9ccc-359044a437c3 admin admin] The token cache is being invalidate because project fa19733707044f81ad9abc87e29b9df3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:55.651466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-12bffacb-22f5-4a90-9ccc-359044a437c3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:55.652201 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-12bffacb-22f5-4a90-9ccc-359044a437c3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:55.654016 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1309/2619] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:55 2022] DELETE /identity/v3/projects/fa19733707044f81ad9abc87e29b9df3 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:55.659344 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2620a946-92b1-4cb6-a7a3-6a9b0f7c2842 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:55.659718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2620a946-92b1-4cb6-a7a3-6a9b0f7c2842 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:55.660063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2620a946-92b1-4cb6-a7a3-6a9b0f7c2842 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:55.691831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-2620a946-92b1-4cb6-a7a3-6a9b0f7c2842 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:55.693322 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-2620a946-92b1-4cb6-a7a3-6a9b0f7c2842 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:55.766163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:55.785388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1311/2620] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:55.785528 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23820bf5-450d-4ecd-ad68-3f5ad0daf12e None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:55.785800 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23820bf5-450d-4ecd-ad68-3f5ad0daf12e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:55.786071 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23820bf5-450d-4ecd-ad68-3f5ad0daf12e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:55.813425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-23820bf5-450d-4ecd-ad68-3f5ad0daf12e None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:55.814649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-23820bf5-450d-4ecd-ad68-3f5ad0daf12e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:55.823402 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1310/2621] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:55.827754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2620a946-92b1-4cb6-a7a3-6a9b0f7c2842 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:55.839491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2620a946-92b1-4cb6-a7a3-6a9b0f7c2842 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:55.844296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a6c356-c119-40bc-86a2-2fe2c88d8d86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:55.844955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a6c356-c119-40bc-86a2-2fe2c88d8d86 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:55.845236 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a6c356-c119-40bc-86a2-2fe2c88d8d86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:55.845516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a6c356-c119-40bc-86a2-2fe2c88d8d86 admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:55.846167 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a6c356-c119-40bc-86a2-2fe2c88d8d86 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:37:55.846467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a6c356-c119-40bc-86a2-2fe2c88d8d86 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:55.848077 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a6c356-c119-40bc-86a2-2fe2c88d8d86 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:55.861710 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1312/2622] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:37:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:37:55.866319 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23820bf5-450d-4ecd-ad68-3f5ad0daf12e None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:55.875518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23820bf5-450d-4ecd-ad68-3f5ad0daf12e None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:55.880142 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cfc090-5d2d-4641-923a-1205ffa3d480 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:55.880627 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cfc090-5d2d-4641-923a-1205ffa3d480 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:55.880901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cfc090-5d2d-4641-923a-1205ffa3d480 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:55.881228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cfc090-5d2d-4641-923a-1205ffa3d480 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:55.881853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cfc090-5d2d-4641-923a-1205ffa3d480 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:55.883725 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cfc090-5d2d-4641-923a-1205ffa3d480 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:55.924272 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1311/2623] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:55 2022] POST /identity/v3/projects => generated 387 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:55.925900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a6c356-c119-40bc-86a2-2fe2c88d8d86 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:55.937549 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a6c356-c119-40bc-86a2-2fe2c88d8d86 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:55.985424 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cfc090-5d2d-4641-923a-1205ffa3d480 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:55.987410 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-00cfc090-5d2d-4641-923a-1205ffa3d480 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:56.014750 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253aba19-3e8a-4cde-96b9-61ad70d1a160 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:56.015779 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253aba19-3e8a-4cde-96b9-61ad70d1a160 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:56.016407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253aba19-3e8a-4cde-96b9-61ad70d1a160 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:56.016978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253aba19-3e8a-4cde-96b9-61ad70d1a160 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:56.018660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253aba19-3e8a-4cde-96b9-61ad70d1a160 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:56.023266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253aba19-3e8a-4cde-96b9-61ad70d1a160 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:56.065774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1313/2624] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:55 2022] POST /identity/v3/users => generated 294 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:56.073386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253aba19-3e8a-4cde-96b9-61ad70d1a160 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:56.082114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cfc090-5d2d-4641-923a-1205ffa3d480 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:56.086743 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253aba19-3e8a-4cde-96b9-61ad70d1a160 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:56.092547 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a877205-43f6-45ca-99d2-b9d91b90580c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:56.093206 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a877205-43f6-45ca-99d2-b9d91b90580c admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:56.095093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:56.095984 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:56.096261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:56.096556 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd admin admin] PATH_INFO: `/v3/projects/17d4f187d45246e68f294e93a0215c7d` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:56.097190 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a877205-43f6-45ca-99d2-b9d91b90580c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:56.097495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a877205-43f6-45ca-99d2-b9d91b90580c admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:56.098172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a877205-43f6-45ca-99d2-b9d91b90580c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:56.100399 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a877205-43f6-45ca-99d2-b9d91b90580c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:56.101722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd admin admin] RBAC: Authorizing `identity:delete_project(project_id=17d4f187d45246e68f294e93a0215c7d)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:56.103672 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:56.136281 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1314/2625] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:56 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:56.140924 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a877205-43f6-45ca-99d2-b9d91b90580c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:56.161001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a877205-43f6-45ca-99d2-b9d91b90580c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:56.163406 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '17d4f187d45246e68f294e93a0215c7d', 'request_id': 'req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:56.170019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '17d4f187d45246e68f294e93a0215c7d', 'request_id': 'req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:56.174624 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:56.177407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '17d4f187d45246e68f294e93a0215c7d', 'request_id': 'req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:56.196280 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd admin admin] The token cache is being invalidate because project 17d4f187d45246e68f294e93a0215c7d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:56.196669 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:56.197304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:56.212634 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1312/2626] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:55 2022] DELETE /identity/v3/projects/17d4f187d45246e68f294e93a0215c7d => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:37:56.251243 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e760936-70d4-4aff-ac29-874da4fe854e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:56.251885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e760936-70d4-4aff-ac29-874da4fe854e admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:56.252162 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e760936-70d4-4aff-ac29-874da4fe854e admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:56.252537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e760936-70d4-4aff-ac29-874da4fe854e admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:56.253193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e760936-70d4-4aff-ac29-874da4fe854e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:56.254866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e760936-70d4-4aff-ac29-874da4fe854e admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:56.273665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1315/2627] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:56 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:37:56.275565 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:56.277703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:56.295202 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1cbdc1-98a5-4b4b-94ff-decadb319ebd admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:56.301435 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac48c0a-d82d-4a8b-94be-16c7252a9f61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:56.302130 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac48c0a-d82d-4a8b-94be-16c7252a9f61 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:56.302406 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac48c0a-d82d-4a8b-94be-16c7252a9f61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:56.302683 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac48c0a-d82d-4a8b-94be-16c7252a9f61 admin admin] PATH_INFO: `/v3/projects/757ae47e0392417a8e725d1c67670f6b/users/131e0d292bbe44e08b2a5c5691178ed8/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:56.320693 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac48c0a-d82d-4a8b-94be-16c7252a9f61 admin admin] RBAC: Authorizing `identity:create_grant(project_id=757ae47e0392417a8e725d1c67670f6b, user_id=131e0d292bbe44e08b2a5c5691178ed8, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:56.323447 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac48c0a-d82d-4a8b-94be-16c7252a9f61 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:56.339550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1313/2628] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:56 2022] PUT /identity/v3/projects/757ae47e0392417a8e725d1c67670f6b/users/131e0d292bbe44e08b2a5c5691178ed8/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:56.352352 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9617686-5a49-46fd-90f6-828fa9a15593 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:56.352634 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9617686-5a49-46fd-90f6-828fa9a15593 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:56.352907 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9617686-5a49-46fd-90f6-828fa9a15593 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:56.395879 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-c9617686-5a49-46fd-90f6-828fa9a15593 None None] MFA Rules not processed for user `131e0d292bbe44e08b2a5c5691178ed8`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:56.397920 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c9617686-5a49-46fd-90f6-828fa9a15593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:56.457330 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:56.491339 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1316/2629] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:56 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:56.496153 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac48c0a-d82d-4a8b-94be-16c7252a9f61 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:56.567909 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac48c0a-d82d-4a8b-94be-16c7252a9f61 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:56.573283 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f146aeae-f4e8-47b6-87de-bcedf70c3b05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:56.573895 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f146aeae-f4e8-47b6-87de-bcedf70c3b05 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:56.574169 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f146aeae-f4e8-47b6-87de-bcedf70c3b05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:56.574436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f146aeae-f4e8-47b6-87de-bcedf70c3b05 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:56.575064 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f146aeae-f4e8-47b6-87de-bcedf70c3b05 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:56.577009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f146aeae-f4e8-47b6-87de-bcedf70c3b05 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:56.609859 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1314/2630] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:56 2022] POST /identity/v3/projects => generated 387 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:56.611110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9617686-5a49-46fd-90f6-828fa9a15593 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:56.623864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9617686-5a49-46fd-90f6-828fa9a15593 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:56.704505 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8854c3-f09c-435b-9899-001d81b3ebf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:56.705141 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8854c3-f09c-435b-9899-001d81b3ebf6 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:56.705419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8854c3-f09c-435b-9899-001d81b3ebf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:56.705697 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8854c3-f09c-435b-9899-001d81b3ebf6 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:56.706346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8854c3-f09c-435b-9899-001d81b3ebf6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:56.708304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8854c3-f09c-435b-9899-001d81b3ebf6 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:56.734341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1317/2631] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:56 2022] POST /identity/v3/users => generated 300 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:56.738535 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f146aeae-f4e8-47b6-87de-bcedf70c3b05 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:56.749745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f146aeae-f4e8-47b6-87de-bcedf70c3b05 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:56.754915 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dddc54-4c5f-4351-84b1-2369e5d0999a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:56.756141 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dddc54-4c5f-4351-84b1-2369e5d0999a admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:56.756867 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dddc54-4c5f-4351-84b1-2369e5d0999a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:56.757322 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dddc54-4c5f-4351-84b1-2369e5d0999a admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:56.760126 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dddc54-4c5f-4351-84b1-2369e5d0999a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:56.760126 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dddc54-4c5f-4351-84b1-2369e5d0999a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:56.779887 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1315/2632] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:56 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:37:56.781304 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8854c3-f09c-435b-9899-001d81b3ebf6 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:56.794264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8854c3-f09c-435b-9899-001d81b3ebf6 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:56.799101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3030da45-2c56-47d9-a8dc-52f3862e066d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:56.799765 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3030da45-2c56-47d9-a8dc-52f3862e066d admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:56.800049 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3030da45-2c56-47d9-a8dc-52f3862e066d admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:56.800356 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3030da45-2c56-47d9-a8dc-52f3862e066d admin admin] PATH_INFO: `/v3/projects/28e89b676dcd4a88bbe1f46d97d8ce1b/users/e1e2b38c5e7144dbb6208a832d386bd8/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:56.816736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3030da45-2c56-47d9-a8dc-52f3862e066d admin admin] RBAC: Authorizing `identity:create_grant(project_id=28e89b676dcd4a88bbe1f46d97d8ce1b, user_id=e1e2b38c5e7144dbb6208a832d386bd8, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:56.819647 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3030da45-2c56-47d9-a8dc-52f3862e066d admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:56.833382 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1318/2633] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:56 2022] PUT /identity/v3/projects/28e89b676dcd4a88bbe1f46d97d8ce1b/users/e1e2b38c5e7144dbb6208a832d386bd8/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:56.838414 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dddc54-4c5f-4351-84b1-2369e5d0999a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:56.849675 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dddc54-4c5f-4351-84b1-2369e5d0999a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:56.914478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a9378d-bb47-49fd-b586-ebed633c2a05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:56.915085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a9378d-bb47-49fd-b586-ebed633c2a05 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:56.915363 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a9378d-bb47-49fd-b586-ebed633c2a05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:56.915635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a9378d-bb47-49fd-b586-ebed633c2a05 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:56.916273 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a9378d-bb47-49fd-b586-ebed633c2a05 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:56.917889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a9378d-bb47-49fd-b586-ebed633c2a05 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:56.928224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1316/2634] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:56 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:56.932404 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3030da45-2c56-47d9-a8dc-52f3862e066d admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:56.943228 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3030da45-2c56-47d9-a8dc-52f3862e066d admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:56.949765 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb0a69f-9354-4107-a60b-eb43e1e5d869 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:56.949952 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb0a69f-9354-4107-a60b-eb43e1e5d869 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:56.950312 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb0a69f-9354-4107-a60b-eb43e1e5d869 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:56.950630 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb0a69f-9354-4107-a60b-eb43e1e5d869 admin admin] PATH_INFO: `/v3/projects/28e89b676dcd4a88bbe1f46d97d8ce1b/users/e1e2b38c5e7144dbb6208a832d386bd8/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:56.952672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb0a69f-9354-4107-a60b-eb43e1e5d869 admin admin] RBAC: Authorizing `identity:create_grant(project_id=28e89b676dcd4a88bbe1f46d97d8ce1b, user_id=e1e2b38c5e7144dbb6208a832d386bd8, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:56.955718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb0a69f-9354-4107-a60b-eb43e1e5d869 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:56.970943 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1319/2635] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:56 2022] PUT /identity/v3/projects/28e89b676dcd4a88bbe1f46d97d8ce1b/users/e1e2b38c5e7144dbb6208a832d386bd8/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:37:56.981186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9043708-502e-41bd-8bdc-84c17c6db39e None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:56.981584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9043708-502e-41bd-8bdc-84c17c6db39e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:56.982282 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9043708-502e-41bd-8bdc-84c17c6db39e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:57.042599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-f9043708-502e-41bd-8bdc-84c17c6db39e None None] MFA Rules not processed for user `e1e2b38c5e7144dbb6208a832d386bd8`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:57.043741 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f9043708-502e-41bd-8bdc-84c17c6db39e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:57.096158 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:DELETE=2,keystone:SELECT=95,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:57.195223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1317/2636] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:56 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:57.207979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb0a69f-9354-4107-a60b-eb43e1e5d869 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:57.210561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9cb0a69f-9354-4107-a60b-eb43e1e5d869 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:57.278680 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb0a69f-9354-4107-a60b-eb43e1e5d869 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:57.282722 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0a2b80-050a-4d5d-867b-5797dbf0f0d9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:57.283461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0a2b80-050a-4d5d-867b-5797dbf0f0d9 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:57.283822 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0a2b80-050a-4d5d-867b-5797dbf0f0d9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:57.284208 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0a2b80-050a-4d5d-867b-5797dbf0f0d9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:57.295624 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0a2b80-050a-4d5d-867b-5797dbf0f0d9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:57.297164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0a2b80-050a-4d5d-867b-5797dbf0f0d9 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:57.309794 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1320/2637] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:57 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:58.008453 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9043708-502e-41bd-8bdc-84c17c6db39e None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:58.082352 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9043708-502e-41bd-8bdc-84c17c6db39e None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:58.087778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe94e03-1465-42b6-8510-3b21bbc7e0eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:58.088443 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe94e03-1465-42b6-8510-3b21bbc7e0eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:58.088722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe94e03-1465-42b6-8510-3b21bbc7e0eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:58.088991 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe94e03-1465-42b6-8510-3b21bbc7e0eb admin admin] PATH_INFO: `/v3/users/131e0d292bbe44e08b2a5c5691178ed8` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:58.090103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe94e03-1465-42b6-8510-3b21bbc7e0eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=131e0d292bbe44e08b2a5c5691178ed8)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:58.092028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe94e03-1465-42b6-8510-3b21bbc7e0eb admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:58.157891 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9fe94e03-1465-42b6-8510-3b21bbc7e0eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '131e0d292bbe44e08b2a5c5691178ed8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:58.167731 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9fe94e03-1465-42b6-8510-3b21bbc7e0eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '131e0d292bbe44e08b2a5c5691178ed8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:58.183890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9fe94e03-1465-42b6-8510-3b21bbc7e0eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '131e0d292bbe44e08b2a5c5691178ed8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:58.204875 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9fe94e03-1465-42b6-8510-3b21bbc7e0eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '131e0d292bbe44e08b2a5c5691178ed8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:58.205748 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9fe94e03-1465-42b6-8510-3b21bbc7e0eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '131e0d292bbe44e08b2a5c5691178ed8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:58.223636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1318/2638] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:58 2022] DELETE /identity/v3/users/131e0d292bbe44e08b2a5c5691178ed8 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:58.228164 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63177c9-5304-4e5c-88e6-8025902cff0d None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:58.228726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63177c9-5304-4e5c-88e6-8025902cff0d None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:58.228726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63177c9-5304-4e5c-88e6-8025902cff0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:58.249363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-d63177c9-5304-4e5c-88e6-8025902cff0d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:58.250702 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d63177c9-5304-4e5c-88e6-8025902cff0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:58.311422 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:58.332965 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1321/2639] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:58.343597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe94e03-1465-42b6-8510-3b21bbc7e0eb admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:58.345041 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9fe94e03-1465-42b6-8510-3b21bbc7e0eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:58.415503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:58.434452 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe94e03-1465-42b6-8510-3b21bbc7e0eb admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:58.438228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd829ff-25eb-4551-91d7-ec3793042f7c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:58.438908 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd829ff-25eb-4551-91d7-ec3793042f7c service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:58.439256 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd829ff-25eb-4551-91d7-ec3793042f7c service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:58.439596 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd829ff-25eb-4551-91d7-ec3793042f7c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:58.449040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd829ff-25eb-4551-91d7-ec3793042f7c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:58.450477 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd829ff-25eb-4551-91d7-ec3793042f7c service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:58.461432 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1319/2640] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:37:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:37:58.756351 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63177c9-5304-4e5c-88e6-8025902cff0d None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:58.759758 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d63177c9-5304-4e5c-88e6-8025902cff0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:58.779231 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63177c9-5304-4e5c-88e6-8025902cff0d None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:58.785879 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028f8041-1d32-4c0e-9ddc-c6750385c9af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:58.786689 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028f8041-1d32-4c0e-9ddc-c6750385c9af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:58.787105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028f8041-1d32-4c0e-9ddc-c6750385c9af admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:58.787935 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028f8041-1d32-4c0e-9ddc-c6750385c9af admin admin] PATH_INFO: `/v3/projects/757ae47e0392417a8e725d1c67670f6b` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:58.789430 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028f8041-1d32-4c0e-9ddc-c6750385c9af admin admin] RBAC: Authorizing `identity:delete_project(project_id=757ae47e0392417a8e725d1c67670f6b)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:58.791616 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028f8041-1d32-4c0e-9ddc-c6750385c9af admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:58.854363 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-028f8041-1d32-4c0e-9ddc-c6750385c9af admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '757ae47e0392417a8e725d1c67670f6b', 'request_id': 'req-028f8041-1d32-4c0e-9ddc-c6750385c9af'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:58.862039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-028f8041-1d32-4c0e-9ddc-c6750385c9af admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '757ae47e0392417a8e725d1c67670f6b', 'request_id': 'req-028f8041-1d32-4c0e-9ddc-c6750385c9af'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:58.870136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-028f8041-1d32-4c0e-9ddc-c6750385c9af admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '757ae47e0392417a8e725d1c67670f6b', 'request_id': 'req-028f8041-1d32-4c0e-9ddc-c6750385c9af'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:58.887454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-028f8041-1d32-4c0e-9ddc-c6750385c9af admin admin] The token cache is being invalidate because project 757ae47e0392417a8e725d1c67670f6b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:58.887752 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-028f8041-1d32-4c0e-9ddc-c6750385c9af admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:58.888491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-028f8041-1d32-4c0e-9ddc-c6750385c9af admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:58.890062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1322/2641] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:58 2022] DELETE /identity/v3/projects/757ae47e0392417a8e725d1c67670f6b => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:58.894536 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd829ff-25eb-4551-91d7-ec3793042f7c service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:58.905012 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd829ff-25eb-4551-91d7-ec3793042f7c service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:58.979197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563c876e-181b-460e-b0e2-d95880b83ec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:58.980333 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c876e-181b-460e-b0e2-d95880b83ec0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:58.980664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c876e-181b-460e-b0e2-d95880b83ec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:58.981061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c876e-181b-460e-b0e2-d95880b83ec0 admin admin] PATH_INFO: `/v3/users/e1e2b38c5e7144dbb6208a832d386bd8` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:58.982370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563c876e-181b-460e-b0e2-d95880b83ec0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1e2b38c5e7144dbb6208a832d386bd8)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:58.984736 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563c876e-181b-460e-b0e2-d95880b83ec0 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:59.042022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-563c876e-181b-460e-b0e2-d95880b83ec0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1e2b38c5e7144dbb6208a832d386bd8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:59.050228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-563c876e-181b-460e-b0e2-d95880b83ec0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1e2b38c5e7144dbb6208a832d386bd8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:59.059350 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-563c876e-181b-460e-b0e2-d95880b83ec0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1e2b38c5e7144dbb6208a832d386bd8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:59.072803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-563c876e-181b-460e-b0e2-d95880b83ec0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1e2b38c5e7144dbb6208a832d386bd8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:59.073549 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-563c876e-181b-460e-b0e2-d95880b83ec0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1e2b38c5e7144dbb6208a832d386bd8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:59.079832 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1320/2642] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:37:58 2022] DELETE /identity/v3/users/e1e2b38c5e7144dbb6208a832d386bd8 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:59.374550 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028f8041-1d32-4c0e-9ddc-c6750385c9af admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:59.376001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-028f8041-1d32-4c0e-9ddc-c6750385c9af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:59.389866 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028f8041-1d32-4c0e-9ddc-c6750385c9af admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:59.451149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23aa129-dc7c-4e76-816f-a609faeefe82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:59.451798 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23aa129-dc7c-4e76-816f-a609faeefe82 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:59.452080 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23aa129-dc7c-4e76-816f-a609faeefe82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:59.452403 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23aa129-dc7c-4e76-816f-a609faeefe82 admin admin] PATH_INFO: `/v3/projects/28e89b676dcd4a88bbe1f46d97d8ce1b` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:59.453556 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23aa129-dc7c-4e76-816f-a609faeefe82 admin admin] RBAC: Authorizing `identity:delete_project(project_id=28e89b676dcd4a88bbe1f46d97d8ce1b)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:59.455852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23aa129-dc7c-4e76-816f-a609faeefe82 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:59.509530 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b23aa129-dc7c-4e76-816f-a609faeefe82 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '28e89b676dcd4a88bbe1f46d97d8ce1b', 'request_id': 'req-b23aa129-dc7c-4e76-816f-a609faeefe82'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:59.515876 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b23aa129-dc7c-4e76-816f-a609faeefe82 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '28e89b676dcd4a88bbe1f46d97d8ce1b', 'request_id': 'req-b23aa129-dc7c-4e76-816f-a609faeefe82'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:59.522270 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b23aa129-dc7c-4e76-816f-a609faeefe82 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '28e89b676dcd4a88bbe1f46d97d8ce1b', 'request_id': 'req-b23aa129-dc7c-4e76-816f-a609faeefe82'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:59.531736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:59.536650 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b23aa129-dc7c-4e76-816f-a609faeefe82 admin admin] The token cache is being invalidate because project 28e89b676dcd4a88bbe1f46d97d8ce1b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:37:59.536964 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b23aa129-dc7c-4e76-816f-a609faeefe82 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:59.538640 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b23aa129-dc7c-4e76-816f-a609faeefe82 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:37:59.540211 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1323/2643] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:37:59 2022] DELETE /identity/v3/projects/28e89b676dcd4a88bbe1f46d97d8ce1b => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:37:59.545327 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20db652b-657a-4342-85ca-4f30e43515ab None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:59.545616 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20db652b-657a-4342-85ca-4f30e43515ab None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:59.545892 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20db652b-657a-4342-85ca-4f30e43515ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:59.565422 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-20db652b-657a-4342-85ca-4f30e43515ab None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:59.566650 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-20db652b-657a-4342-85ca-4f30e43515ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:59.620732 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:37:59.630088 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1321/2644] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:59.639108 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3c9e5a-b9ac-4f69-ac5c-3bb96672c072 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:59.639369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3c9e5a-b9ac-4f69-ac5c-3bb96672c072 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:59.639647 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3c9e5a-b9ac-4f69-ac5c-3bb96672c072 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:59.660234 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-5f3c9e5a-b9ac-4f69-ac5c-3bb96672c072 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:37:59.661470 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-5f3c9e5a-b9ac-4f69-ac5c-3bb96672c072 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:37:59.670113 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1324/2645] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:37:59.674576 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20db652b-657a-4342-85ca-4f30e43515ab None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:59.684329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20db652b-657a-4342-85ca-4f30e43515ab None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:59.688840 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429f1e8f-d02e-4989-af19-25f735e55b35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:59.689432 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429f1e8f-d02e-4989-af19-25f735e55b35 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:59.689710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429f1e8f-d02e-4989-af19-25f735e55b35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:59.689983 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429f1e8f-d02e-4989-af19-25f735e55b35 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:59.690622 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429f1e8f-d02e-4989-af19-25f735e55b35 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:37:59.690924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429f1e8f-d02e-4989-af19-25f735e55b35 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:59.692528 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429f1e8f-d02e-4989-af19-25f735e55b35 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:59.704663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1322/2646] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:37:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:37:59.709707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3c9e5a-b9ac-4f69-ac5c-3bb96672c072 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:59.719303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3c9e5a-b9ac-4f69-ac5c-3bb96672c072 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:59.723979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1639f35f-ef95-46b4-a6cd-cbb2a03420d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:59.725410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1639f35f-ef95-46b4-a6cd-cbb2a03420d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:59.725694 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1639f35f-ef95-46b4-a6cd-cbb2a03420d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:59.726121 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1639f35f-ef95-46b4-a6cd-cbb2a03420d7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:59.726762 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1639f35f-ef95-46b4-a6cd-cbb2a03420d7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:59.728716 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1639f35f-ef95-46b4-a6cd-cbb2a03420d7 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:59.756993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1325/2647] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:37:59 2022] POST /identity/v3/projects => generated 381 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:59.761227 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429f1e8f-d02e-4989-af19-25f735e55b35 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:59.771263 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429f1e8f-d02e-4989-af19-25f735e55b35 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:59.826735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8a9dc5-b95d-48b4-9487-7e82a189917a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:59.827329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8a9dc5-b95d-48b4-9487-7e82a189917a admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:59.827604 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8a9dc5-b95d-48b4-9487-7e82a189917a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:59.827870 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8a9dc5-b95d-48b4-9487-7e82a189917a admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:59.828510 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8a9dc5-b95d-48b4-9487-7e82a189917a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:59.830390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8a9dc5-b95d-48b4-9487-7e82a189917a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:59.853328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1323/2648] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:37:59 2022] POST /identity/v3/users => generated 291 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:37:59.857792 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1639f35f-ef95-46b4-a6cd-cbb2a03420d7 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:59.868972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1639f35f-ef95-46b4-a6cd-cbb2a03420d7 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:59.874144 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b9df1c-4b96-455b-b9be-1c8342904a73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:59.874688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b9df1c-4b96-455b-b9be-1c8342904a73 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:59.874970 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b9df1c-4b96-455b-b9be-1c8342904a73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:59.875245 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b9df1c-4b96-455b-b9be-1c8342904a73 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:59.875925 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b9df1c-4b96-455b-b9be-1c8342904a73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:59.877699 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b9df1c-4b96-455b-b9be-1c8342904a73 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:59.889081 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1326/2649] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:59 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:59.893152 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8a9dc5-b95d-48b4-9487-7e82a189917a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:59.904271 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8a9dc5-b95d-48b4-9487-7e82a189917a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:59.908895 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d51362-670f-4c2b-954d-75503799822e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:59.909491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d51362-670f-4c2b-954d-75503799822e admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:59.909772 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d51362-670f-4c2b-954d-75503799822e admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:59.910043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d51362-670f-4c2b-954d-75503799822e admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:59.910678 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d51362-670f-4c2b-954d-75503799822e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:59.912262 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d51362-670f-4c2b-954d-75503799822e admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:59.924682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1324/2650] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:37:59 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:37:59.928984 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b9df1c-4b96-455b-b9be-1c8342904a73 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:37:59.939129 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b9df1c-4b96-455b-b9be-1c8342904a73 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:37:59.944463 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283242ea-8e87-466d-a4ea-c0ea9e4e47eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:37:59.945110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283242ea-8e87-466d-a4ea-c0ea9e4e47eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:59.945393 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283242ea-8e87-466d-a4ea-c0ea9e4e47eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:59.945667 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283242ea-8e87-466d-a4ea-c0ea9e4e47eb admin admin] PATH_INFO: `/v3/projects/890c34001ed64337a4bb61e9aa84876f/users/8d7e61c6aa1d4f448f4b5c4e14fc76cc/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:37:59.963244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283242ea-8e87-466d-a4ea-c0ea9e4e47eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=890c34001ed64337a4bb61e9aa84876f, user_id=8d7e61c6aa1d4f448f4b5c4e14fc76cc, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:37:59.965948 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283242ea-8e87-466d-a4ea-c0ea9e4e47eb admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:37:59.980262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1327/2651] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:37:59 2022] PUT /identity/v3/projects/890c34001ed64337a4bb61e9aa84876f/users/8d7e61c6aa1d4f448f4b5c4e14fc76cc/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:37:59.991728 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f130bd2-73bc-4837-a084-fa488bb722a2 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:37:59.992019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f130bd2-73bc-4837-a084-fa488bb722a2 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:37:59.992288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f130bd2-73bc-4837-a084-fa488bb722a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:00.022198 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-9f130bd2-73bc-4837-a084-fa488bb722a2 None None] MFA Rules not processed for user `8d7e61c6aa1d4f448f4b5c4e14fc76cc`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:00.026545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9f130bd2-73bc-4837-a084-fa488bb722a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:00.093964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1325/2652] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:37:59 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:00.098506 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283242ea-8e87-466d-a4ea-c0ea9e4e47eb admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:00.180478 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283242ea-8e87-466d-a4ea-c0ea9e4e47eb admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:00.185826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7c92e5-8bda-45c6-84af-be769ecbeacf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:00.186503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7c92e5-8bda-45c6-84af-be769ecbeacf admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:00.186840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7c92e5-8bda-45c6-84af-be769ecbeacf admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:00.187120 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7c92e5-8bda-45c6-84af-be769ecbeacf admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:00.187754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7c92e5-8bda-45c6-84af-be769ecbeacf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:00.189691 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7c92e5-8bda-45c6-84af-be769ecbeacf admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:00.237838 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1328/2653] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:00 2022] POST /identity/v3/projects => generated 379 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:00.245080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f130bd2-73bc-4837-a084-fa488bb722a2 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:00.258403 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f130bd2-73bc-4837-a084-fa488bb722a2 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:00.310951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:00.319309 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685aaf19-7019-474a-875e-e516e58464e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:00.319910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685aaf19-7019-474a-875e-e516e58464e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:00.320184 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685aaf19-7019-474a-875e-e516e58464e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:00.320497 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685aaf19-7019-474a-875e-e516e58464e1 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:00.321115 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685aaf19-7019-474a-875e-e516e58464e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:00.322996 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685aaf19-7019-474a-875e-e516e58464e1 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:00.344207 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1326/2654] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:00 2022] POST /identity/v3/users => generated 296 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:00.347970 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7c92e5-8bda-45c6-84af-be769ecbeacf admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:00.359565 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7c92e5-8bda-45c6-84af-be769ecbeacf admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:00.364203 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b54dd1d-479e-4e56-837c-4871c7589eb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:00.364830 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b54dd1d-479e-4e56-837c-4871c7589eb6 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:00.365719 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b54dd1d-479e-4e56-837c-4871c7589eb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:00.366159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b54dd1d-479e-4e56-837c-4871c7589eb6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:00.366805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b54dd1d-479e-4e56-837c-4871c7589eb6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:00.368447 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b54dd1d-479e-4e56-837c-4871c7589eb6 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:00.379571 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1329/2655] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:00 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:00.385428 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685aaf19-7019-474a-875e-e516e58464e1 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:00.395764 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685aaf19-7019-474a-875e-e516e58464e1 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:00.401174 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410c83ba-2b50-4388-9f7b-194496573f1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:00.401751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410c83ba-2b50-4388-9f7b-194496573f1c admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:00.402032 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410c83ba-2b50-4388-9f7b-194496573f1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:00.402594 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410c83ba-2b50-4388-9f7b-194496573f1c admin admin] PATH_INFO: `/v3/projects/0e61ee100a8c41f5ab8b450e3c50180f/users/b61c753a9ab4450385aa48a84365edad/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:00.420457 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410c83ba-2b50-4388-9f7b-194496573f1c admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e61ee100a8c41f5ab8b450e3c50180f, user_id=b61c753a9ab4450385aa48a84365edad, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:00.423089 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410c83ba-2b50-4388-9f7b-194496573f1c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:00.439167 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1327/2656] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:00 2022] PUT /identity/v3/projects/0e61ee100a8c41f5ab8b450e3c50180f/users/b61c753a9ab4450385aa48a84365edad/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:00.443862 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b54dd1d-479e-4e56-837c-4871c7589eb6 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:00.455511 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b54dd1d-479e-4e56-837c-4871c7589eb6 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:00.529211 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8c679c-56f9-4ace-9843-8565a45dd74c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:00.529831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8c679c-56f9-4ace-9843-8565a45dd74c admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:00.530118 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8c679c-56f9-4ace-9843-8565a45dd74c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:00.530426 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8c679c-56f9-4ace-9843-8565a45dd74c admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:00.531074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8c679c-56f9-4ace-9843-8565a45dd74c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:00.532782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8c679c-56f9-4ace-9843-8565a45dd74c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:00.550179 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1330/2657] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:00 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:00.551683 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410c83ba-2b50-4388-9f7b-194496573f1c admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:00.562994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410c83ba-2b50-4388-9f7b-194496573f1c admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:00.567908 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7e961d-edf4-4090-b012-b11e3710337b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:00.569639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e961d-edf4-4090-b012-b11e3710337b admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:00.569913 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e961d-edf4-4090-b012-b11e3710337b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:00.570185 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e961d-edf4-4090-b012-b11e3710337b admin admin] PATH_INFO: `/v3/projects/0e61ee100a8c41f5ab8b450e3c50180f/users/b61c753a9ab4450385aa48a84365edad/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:00.572136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7e961d-edf4-4090-b012-b11e3710337b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e61ee100a8c41f5ab8b450e3c50180f, user_id=b61c753a9ab4450385aa48a84365edad, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:00.574813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7e961d-edf4-4090-b012-b11e3710337b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:00.585918 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1328/2658] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:00 2022] PUT /identity/v3/projects/0e61ee100a8c41f5ab8b450e3c50180f/users/b61c753a9ab4450385aa48a84365edad/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:00.596357 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584a275c-4ab2-42fa-829a-5a9a36cac7b4 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:00.596657 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584a275c-4ab2-42fa-829a-5a9a36cac7b4 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:00.596967 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584a275c-4ab2-42fa-829a-5a9a36cac7b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:00.606154 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:00.627827 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-584a275c-4ab2-42fa-829a-5a9a36cac7b4 None None] MFA Rules not processed for user `b61c753a9ab4450385aa48a84365edad`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:00.629130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-584a275c-4ab2-42fa-829a-5a9a36cac7b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:00.704730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1331/2659] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:00 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:00.716088 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7e961d-edf4-4090-b012-b11e3710337b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:00.717822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-4b7e961d-edf4-4090-b012-b11e3710337b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:00.780759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7e961d-edf4-4090-b012-b11e3710337b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:00.784472 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcc0c82-f28b-4aa0-8aef-6f90de00787b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:00.785080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcc0c82-f28b-4aa0-8aef-6f90de00787b service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:00.785357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcc0c82-f28b-4aa0-8aef-6f90de00787b service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:00.785629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcc0c82-f28b-4aa0-8aef-6f90de00787b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:00.795842 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcc0c82-f28b-4aa0-8aef-6f90de00787b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:00.797469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcc0c82-f28b-4aa0-8aef-6f90de00787b service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:00.808296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1329/2660] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:00 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:01.225053 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584a275c-4ab2-42fa-829a-5a9a36cac7b4 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:01.345441 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584a275c-4ab2-42fa-829a-5a9a36cac7b4 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:01.362484 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5604fb28-104c-4073-94a7-83c343bb7948 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:01.363350 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5604fb28-104c-4073-94a7-83c343bb7948 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:01.363641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5604fb28-104c-4073-94a7-83c343bb7948 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:01.363963 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5604fb28-104c-4073-94a7-83c343bb7948 admin admin] PATH_INFO: `/v3/users/8d7e61c6aa1d4f448f4b5c4e14fc76cc` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:01.367742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5604fb28-104c-4073-94a7-83c343bb7948 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8d7e61c6aa1d4f448f4b5c4e14fc76cc)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:01.369921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5604fb28-104c-4073-94a7-83c343bb7948 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:01.411146 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcc0c82-f28b-4aa0-8aef-6f90de00787b service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:01.414144 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-adcc0c82-f28b-4aa0-8aef-6f90de00787b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:01.496886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5604fb28-104c-4073-94a7-83c343bb7948 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8d7e61c6aa1d4f448f4b5c4e14fc76cc'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:01.499990 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5604fb28-104c-4073-94a7-83c343bb7948 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8d7e61c6aa1d4f448f4b5c4e14fc76cc'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:01.501217 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5604fb28-104c-4073-94a7-83c343bb7948 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8d7e61c6aa1d4f448f4b5c4e14fc76cc'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:01.511677 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5604fb28-104c-4073-94a7-83c343bb7948 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8d7e61c6aa1d4f448f4b5c4e14fc76cc'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:01.529889 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5604fb28-104c-4073-94a7-83c343bb7948 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8d7e61c6aa1d4f448f4b5c4e14fc76cc'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:01.540010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcc0c82-f28b-4aa0-8aef-6f90de00787b service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:01.545043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d914f3b-39d3-4640-85ac-7ba6f24c856f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:01.545657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d914f3b-39d3-4640-85ac-7ba6f24c856f service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:01.545983 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d914f3b-39d3-4640-85ac-7ba6f24c856f service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:01.546255 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d914f3b-39d3-4640-85ac-7ba6f24c856f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:01.548026 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9d914f3b-39d3-4640-85ac-7ba6f24c856f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:01.549561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1332/2661] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:01 2022] DELETE /identity/v3/users/8d7e61c6aa1d4f448f4b5c4e14fc76cc => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:01.554649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89db4033-c5a9-4329-a5d6-b2864646564d None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:01.554996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89db4033-c5a9-4329-a5d6-b2864646564d None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:01.555350 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89db4033-c5a9-4329-a5d6-b2864646564d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:01.569080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d914f3b-39d3-4640-85ac-7ba6f24c856f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:01.570999 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d914f3b-39d3-4640-85ac-7ba6f24c856f service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:01.584375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1330/2662] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:38:01.605445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-89db4033-c5a9-4329-a5d6-b2864646564d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:01.608072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-89db4033-c5a9-4329-a5d6-b2864646564d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:01.613139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:01.684919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1333/2663] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:38:01.696368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d914f3b-39d3-4640-85ac-7ba6f24c856f service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:01.697863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9d914f3b-39d3-4640-85ac-7ba6f24c856f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:01.710908 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d914f3b-39d3-4640-85ac-7ba6f24c856f service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:01.773022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d9b555-a875-4436-87ae-dbe4627be462 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:01.773784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d9b555-a875-4436-87ae-dbe4627be462 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:01.774065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d9b555-a875-4436-87ae-dbe4627be462 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:01.774333 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d9b555-a875-4436-87ae-dbe4627be462 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:01.782060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:01.785149 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d9b555-a875-4436-87ae-dbe4627be462 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:01.788283 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d9b555-a875-4436-87ae-dbe4627be462 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:01.800658 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1331/2664] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:38:02.174895 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89db4033-c5a9-4329-a5d6-b2864646564d None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:02.176473 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-89db4033-c5a9-4329-a5d6-b2864646564d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:02.190890 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89db4033-c5a9-4329-a5d6-b2864646564d None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:02.198053 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c66290-4178-4849-bea4-3b0c0c8c743e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:02.198458 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c66290-4178-4849-bea4-3b0c0c8c743e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:02.198810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c66290-4178-4849-bea4-3b0c0c8c743e admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:02.199178 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c66290-4178-4849-bea4-3b0c0c8c743e admin admin] PATH_INFO: `/v3/projects/890c34001ed64337a4bb61e9aa84876f` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:02.201653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c66290-4178-4849-bea4-3b0c0c8c743e admin admin] RBAC: Authorizing `identity:delete_project(project_id=890c34001ed64337a4bb61e9aa84876f)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:02.203654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c66290-4178-4849-bea4-3b0c0c8c743e admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:02.268173 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-81c66290-4178-4849-bea4-3b0c0c8c743e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '890c34001ed64337a4bb61e9aa84876f', 'request_id': 'req-81c66290-4178-4849-bea4-3b0c0c8c743e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:02.278780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-81c66290-4178-4849-bea4-3b0c0c8c743e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '890c34001ed64337a4bb61e9aa84876f', 'request_id': 'req-81c66290-4178-4849-bea4-3b0c0c8c743e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:02.288137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-81c66290-4178-4849-bea4-3b0c0c8c743e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '890c34001ed64337a4bb61e9aa84876f', 'request_id': 'req-81c66290-4178-4849-bea4-3b0c0c8c743e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:02.304262 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-81c66290-4178-4849-bea4-3b0c0c8c743e admin admin] The token cache is being invalidate because project 890c34001ed64337a4bb61e9aa84876f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:02.304492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-81c66290-4178-4849-bea4-3b0c0c8c743e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:02.305224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-81c66290-4178-4849-bea4-3b0c0c8c743e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:02.307877 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1334/2665] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:02 2022] DELETE /identity/v3/projects/890c34001ed64337a4bb61e9aa84876f => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:02.312182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d9b555-a875-4436-87ae-dbe4627be462 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:02.331206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d9b555-a875-4436-87ae-dbe4627be462 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:02.399173 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c68af1-a0fa-4f9c-80d6-b55a8870f181 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:02.399804 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c68af1-a0fa-4f9c-80d6-b55a8870f181 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:02.400109 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c68af1-a0fa-4f9c-80d6-b55a8870f181 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:02.400425 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c68af1-a0fa-4f9c-80d6-b55a8870f181 admin admin] PATH_INFO: `/v3/users/b61c753a9ab4450385aa48a84365edad` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:02.401563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c68af1-a0fa-4f9c-80d6-b55a8870f181 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b61c753a9ab4450385aa48a84365edad)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:02.403742 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c68af1-a0fa-4f9c-80d6-b55a8870f181 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:02.462984 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-d5c68af1-a0fa-4f9c-80d6-b55a8870f181 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b61c753a9ab4450385aa48a84365edad'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:02.471997 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-d5c68af1-a0fa-4f9c-80d6-b55a8870f181 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b61c753a9ab4450385aa48a84365edad'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:02.480776 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-d5c68af1-a0fa-4f9c-80d6-b55a8870f181 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b61c753a9ab4450385aa48a84365edad'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:02.493930 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-d5c68af1-a0fa-4f9c-80d6-b55a8870f181 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b61c753a9ab4450385aa48a84365edad'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:02.494665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-d5c68af1-a0fa-4f9c-80d6-b55a8870f181 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b61c753a9ab4450385aa48a84365edad'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:02.500854 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1332/2666] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:02 2022] DELETE /identity/v3/users/b61c753a9ab4450385aa48a84365edad => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:03.077929 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c66290-4178-4849-bea4-3b0c0c8c743e admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:03.080627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-81c66290-4178-4849-bea4-3b0c0c8c743e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:03.094938 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c66290-4178-4849-bea4-3b0c0c8c743e admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:03.169770 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d093eaf4-d08a-445e-b319-405bfd8c1f7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:03.170419 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d093eaf4-d08a-445e-b319-405bfd8c1f7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:03.170703 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d093eaf4-d08a-445e-b319-405bfd8c1f7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:03.170979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d093eaf4-d08a-445e-b319-405bfd8c1f7c admin admin] PATH_INFO: `/v3/projects/0e61ee100a8c41f5ab8b450e3c50180f` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:03.172257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d093eaf4-d08a-445e-b319-405bfd8c1f7c admin admin] RBAC: Authorizing `identity:delete_project(project_id=0e61ee100a8c41f5ab8b450e3c50180f)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:03.174266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d093eaf4-d08a-445e-b319-405bfd8c1f7c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:03.237965 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:03.253255 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d093eaf4-d08a-445e-b319-405bfd8c1f7c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0e61ee100a8c41f5ab8b450e3c50180f', 'request_id': 'req-d093eaf4-d08a-445e-b319-405bfd8c1f7c'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:03.263855 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d093eaf4-d08a-445e-b319-405bfd8c1f7c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0e61ee100a8c41f5ab8b450e3c50180f', 'request_id': 'req-d093eaf4-d08a-445e-b319-405bfd8c1f7c'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:03.274889 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d093eaf4-d08a-445e-b319-405bfd8c1f7c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0e61ee100a8c41f5ab8b450e3c50180f', 'request_id': 'req-d093eaf4-d08a-445e-b319-405bfd8c1f7c'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:03.292753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d093eaf4-d08a-445e-b319-405bfd8c1f7c admin admin] The token cache is being invalidate because project 0e61ee100a8c41f5ab8b450e3c50180f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:03.293061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d093eaf4-d08a-445e-b319-405bfd8c1f7c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:03.293736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d093eaf4-d08a-445e-b319-405bfd8c1f7c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:03.295467 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1335/2667] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:03 2022] DELETE /identity/v3/projects/0e61ee100a8c41f5ab8b450e3c50180f => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:03.300570 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f97790-222b-4adf-925e-20dee1310d34 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:03.300903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f97790-222b-4adf-925e-20dee1310d34 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:03.301186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f97790-222b-4adf-925e-20dee1310d34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:03.326997 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-96f97790-222b-4adf-925e-20dee1310d34 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:03.328266 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-96f97790-222b-4adf-925e-20dee1310d34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:03.445078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1333/2668] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:38:03.453089 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fa73f0-0bf7-461f-a6ac-175379a09aa3 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:03.453377 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fa73f0-0bf7-461f-a6ac-175379a09aa3 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:03.453664 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fa73f0-0bf7-461f-a6ac-175379a09aa3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:03.491595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-e6fa73f0-0bf7-461f-a6ac-175379a09aa3 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:03.493849 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-e6fa73f0-0bf7-461f-a6ac-175379a09aa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:03.503907 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1336/2669] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:03.509042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f97790-222b-4adf-925e-20dee1310d34 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:03.527471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f97790-222b-4adf-925e-20dee1310d34 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:03.534087 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a83261-8d61-4775-9946-2c4f53791f19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:03.534965 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a83261-8d61-4775-9946-2c4f53791f19 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:03.535393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a83261-8d61-4775-9946-2c4f53791f19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:03.536617 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a83261-8d61-4775-9946-2c4f53791f19 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:03.539034 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a83261-8d61-4775-9946-2c4f53791f19 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:38:03.539587 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a83261-8d61-4775-9946-2c4f53791f19 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:03.541426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a83261-8d61-4775-9946-2c4f53791f19 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:03.557733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1334/2670] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:38:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:38:03.562094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fa73f0-0bf7-461f-a6ac-175379a09aa3 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:03.576017 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fa73f0-0bf7-461f-a6ac-175379a09aa3 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:03.583058 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36664e0d-4ccd-443f-bfe6-aded31538e71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:03.584078 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36664e0d-4ccd-443f-bfe6-aded31538e71 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:03.584410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36664e0d-4ccd-443f-bfe6-aded31538e71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:03.584685 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36664e0d-4ccd-443f-bfe6-aded31538e71 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:03.585354 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36664e0d-4ccd-443f-bfe6-aded31538e71 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:03.587727 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36664e0d-4ccd-443f-bfe6-aded31538e71 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:03.634593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1337/2671] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:03 2022] POST /identity/v3/projects => generated 383 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:03.635696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a83261-8d61-4775-9946-2c4f53791f19 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:03.644703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:03.656446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a83261-8d61-4775-9946-2c4f53791f19 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:03.809408 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda2595f-ccc0-4330-88e0-39720657387a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:03.810027 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda2595f-ccc0-4330-88e0-39720657387a admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:03.810307 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda2595f-ccc0-4330-88e0-39720657387a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:03.810574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda2595f-ccc0-4330-88e0-39720657387a admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:03.811337 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda2595f-ccc0-4330-88e0-39720657387a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:03.814034 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda2595f-ccc0-4330-88e0-39720657387a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:03.848452 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1335/2672] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:03 2022] POST /identity/v3/users => generated 292 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:03.852558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36664e0d-4ccd-443f-bfe6-aded31538e71 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:03.868521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36664e0d-4ccd-443f-bfe6-aded31538e71 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:03.888028 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450ea72a-0814-4818-9058-e9cc4e4d61a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:03.888722 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450ea72a-0814-4818-9058-e9cc4e4d61a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:03.889278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450ea72a-0814-4818-9058-e9cc4e4d61a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:03.893692 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450ea72a-0814-4818-9058-e9cc4e4d61a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:03.896247 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450ea72a-0814-4818-9058-e9cc4e4d61a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:03.899343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450ea72a-0814-4818-9058-e9cc4e4d61a0 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:03.922354 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1338/2673] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:03 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:03.928297 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda2595f-ccc0-4330-88e0-39720657387a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:03.938927 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda2595f-ccc0-4330-88e0-39720657387a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:03.944412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48dd3b9-25c6-43a7-8cf6-3e09c024b11b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:03.945550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48dd3b9-25c6-43a7-8cf6-3e09c024b11b admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:03.945550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48dd3b9-25c6-43a7-8cf6-3e09c024b11b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:03.945765 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48dd3b9-25c6-43a7-8cf6-3e09c024b11b admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:03.949387 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48dd3b9-25c6-43a7-8cf6-3e09c024b11b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:03.951093 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48dd3b9-25c6-43a7-8cf6-3e09c024b11b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:03.983329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1336/2674] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:03 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:03.983516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450ea72a-0814-4818-9058-e9cc4e4d61a0 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:03.990518 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450ea72a-0814-4818-9058-e9cc4e4d61a0 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:04.008987 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5199d0d3-d5ce-4050-9d67-5ebe1c805cf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:04.010193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5199d0d3-d5ce-4050-9d67-5ebe1c805cf5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:04.010584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5199d0d3-d5ce-4050-9d67-5ebe1c805cf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:04.010964 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5199d0d3-d5ce-4050-9d67-5ebe1c805cf5 admin admin] PATH_INFO: `/v3/projects/ac15ca31cae8480ca7680a077033e129/users/2384d806893b4f22b22138880a44d4ad/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:04.031938 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5199d0d3-d5ce-4050-9d67-5ebe1c805cf5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac15ca31cae8480ca7680a077033e129, user_id=2384d806893b4f22b22138880a44d4ad, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:04.034837 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5199d0d3-d5ce-4050-9d67-5ebe1c805cf5 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:04.060386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1339/2675] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:03 2022] PUT /identity/v3/projects/ac15ca31cae8480ca7680a077033e129/users/2384d806893b4f22b22138880a44d4ad/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:04.064784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3edee5-21e4-4d2f-bd5c-2f719edb737e None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:04.065063 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3edee5-21e4-4d2f-bd5c-2f719edb737e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:04.065340 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3edee5-21e4-4d2f-bd5c-2f719edb737e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:04.098632 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-3c3edee5-21e4-4d2f-bd5c-2f719edb737e None None] MFA Rules not processed for user `2384d806893b4f22b22138880a44d4ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:04.100374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-3c3edee5-21e4-4d2f-bd5c-2f719edb737e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:04.168274 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1337/2676] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:04 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:04.172771 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5199d0d3-d5ce-4050-9d67-5ebe1c805cf5 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:04.278730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5199d0d3-d5ce-4050-9d67-5ebe1c805cf5 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:04.284358 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346db2c5-0b8a-49b1-b981-8e59d714d4d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:04.284871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346db2c5-0b8a-49b1-b981-8e59d714d4d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:04.285153 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346db2c5-0b8a-49b1-b981-8e59d714d4d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:04.285429 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346db2c5-0b8a-49b1-b981-8e59d714d4d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:04.286077 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346db2c5-0b8a-49b1-b981-8e59d714d4d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:04.288825 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346db2c5-0b8a-49b1-b981-8e59d714d4d4 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:04.351482 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1340/2677] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:04 2022] POST /identity/v3/projects => generated 383 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:38:04.355346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3edee5-21e4-4d2f-bd5c-2f719edb737e None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:04.367840 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3edee5-21e4-4d2f-bd5c-2f719edb737e None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:04.481406 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3c3f3a-975c-4558-a178-771cdac27974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:04.482234 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3c3f3a-975c-4558-a178-771cdac27974 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:04.482610 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3c3f3a-975c-4558-a178-771cdac27974 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:04.482960 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3c3f3a-975c-4558-a178-771cdac27974 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:04.484163 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3c3f3a-975c-4558-a178-771cdac27974 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:04.486490 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3c3f3a-975c-4558-a178-771cdac27974 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:04.507104 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:04.515285 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1338/2678] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:04 2022] POST /identity/v3/users => generated 298 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:04.519060 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346db2c5-0b8a-49b1-b981-8e59d714d4d4 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:04.531410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346db2c5-0b8a-49b1-b981-8e59d714d4d4 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:04.538784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cacc1f-ed55-45f3-ab18-693dba15bb7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:04.539793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cacc1f-ed55-45f3-ab18-693dba15bb7a admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:04.540927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cacc1f-ed55-45f3-ab18-693dba15bb7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:04.541393 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cacc1f-ed55-45f3-ab18-693dba15bb7a admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:04.542244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cacc1f-ed55-45f3-ab18-693dba15bb7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:04.544060 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cacc1f-ed55-45f3-ab18-693dba15bb7a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:04.559796 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1341/2679] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:04 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:04.561089 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3c3f3a-975c-4558-a178-771cdac27974 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:04.573073 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3c3f3a-975c-4558-a178-771cdac27974 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:04.578663 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a6176c-5d97-43cf-8815-82297f79bfc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:04.579717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a6176c-5d97-43cf-8815-82297f79bfc4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:04.580165 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a6176c-5d97-43cf-8815-82297f79bfc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:04.580940 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a6176c-5d97-43cf-8815-82297f79bfc4 admin admin] PATH_INFO: `/v3/projects/a95fe3175a484fd8a3d91f6564460da1/users/57d4996e18674471a33c4486bf6bf4b8/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:04.601759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a6176c-5d97-43cf-8815-82297f79bfc4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a95fe3175a484fd8a3d91f6564460da1, user_id=57d4996e18674471a33c4486bf6bf4b8, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:04.604557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a6176c-5d97-43cf-8815-82297f79bfc4 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:04.623069 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1339/2680] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:04 2022] PUT /identity/v3/projects/a95fe3175a484fd8a3d91f6564460da1/users/57d4996e18674471a33c4486bf6bf4b8/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:04.624432 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cacc1f-ed55-45f3-ab18-693dba15bb7a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:04.636121 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cacc1f-ed55-45f3-ab18-693dba15bb7a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:04.673094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:DELETE=5,keystone:SELECT=91,keystone:INSERT=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:04.705856 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8da5542-6cdc-4125-a5cb-03c6236d0b56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:04.706524 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8da5542-6cdc-4125-a5cb-03c6236d0b56 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:04.706807 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8da5542-6cdc-4125-a5cb-03c6236d0b56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:04.707081 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8da5542-6cdc-4125-a5cb-03c6236d0b56 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:04.707726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8da5542-6cdc-4125-a5cb-03c6236d0b56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:04.709406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8da5542-6cdc-4125-a5cb-03c6236d0b56 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:04.726267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a6176c-5d97-43cf-8815-82297f79bfc4 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:04.739294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1342/2681] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:04 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:04.739481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a6176c-5d97-43cf-8815-82297f79bfc4 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:04.762633 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90df691c-6305-4510-9915-026ce4abbca5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:04.763326 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90df691c-6305-4510-9915-026ce4abbca5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:04.763600 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90df691c-6305-4510-9915-026ce4abbca5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:04.763869 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90df691c-6305-4510-9915-026ce4abbca5 admin admin] PATH_INFO: `/v3/projects/a95fe3175a484fd8a3d91f6564460da1/users/57d4996e18674471a33c4486bf6bf4b8/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:04.766117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90df691c-6305-4510-9915-026ce4abbca5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a95fe3175a484fd8a3d91f6564460da1, user_id=57d4996e18674471a33c4486bf6bf4b8, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:04.768887 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90df691c-6305-4510-9915-026ce4abbca5 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:04.784803 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1340/2682] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:04 2022] PUT /identity/v3/projects/a95fe3175a484fd8a3d91f6564460da1/users/57d4996e18674471a33c4486bf6bf4b8/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:04.866121 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cd5f63-217a-4463-9a4c-c72662e99a38 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:04.866392 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cd5f63-217a-4463-9a4c-c72662e99a38 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:04.866664 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cd5f63-217a-4463-9a4c-c72662e99a38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:04.901699 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-f9cd5f63-217a-4463-9a4c-c72662e99a38 None None] MFA Rules not processed for user `57d4996e18674471a33c4486bf6bf4b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:04.903667 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-f9cd5f63-217a-4463-9a4c-c72662e99a38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:05.008200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1343/2683] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:04 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:05.015756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90df691c-6305-4510-9915-026ce4abbca5 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:05.018713 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-90df691c-6305-4510-9915-026ce4abbca5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:05.114778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90df691c-6305-4510-9915-026ce4abbca5 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:05.119919 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d486992f-618f-4618-9b12-d858aee83237 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:05.120635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d486992f-618f-4618-9b12-d858aee83237 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:05.121142 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d486992f-618f-4618-9b12-d858aee83237 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:05.121503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d486992f-618f-4618-9b12-d858aee83237 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:05.138208 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d486992f-618f-4618-9b12-d858aee83237 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:05.139771 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d486992f-618f-4618-9b12-d858aee83237 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:05.152518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1341/2684] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:05 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:05.291509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cd5f63-217a-4463-9a4c-c72662e99a38 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:05.307013 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cd5f63-217a-4463-9a4c-c72662e99a38 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:05.314706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1290d27-1bc0-4cf6-a80c-66d56847ace7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:05.315346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1290d27-1bc0-4cf6-a80c-66d56847ace7 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:05.315637 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1290d27-1bc0-4cf6-a80c-66d56847ace7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:05.315917 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1290d27-1bc0-4cf6-a80c-66d56847ace7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:05.388569 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1290d27-1bc0-4cf6-a80c-66d56847ace7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:05.390041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1290d27-1bc0-4cf6-a80c-66d56847ace7 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:05.405890 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1344/2685] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:05 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:05.800911 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d486992f-618f-4618-9b12-d858aee83237 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:05.888008 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d486992f-618f-4618-9b12-d858aee83237 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:05.892666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78cab8c-6a3a-4b0f-ba86-44214a5aa0a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:05.893953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78cab8c-6a3a-4b0f-ba86-44214a5aa0a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:05.894434 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78cab8c-6a3a-4b0f-ba86-44214a5aa0a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:05.895420 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78cab8c-6a3a-4b0f-ba86-44214a5aa0a3 admin admin] PATH_INFO: `/v3/users/2384d806893b4f22b22138880a44d4ad` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:05.897114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78cab8c-6a3a-4b0f-ba86-44214a5aa0a3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2384d806893b4f22b22138880a44d4ad)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:05.899679 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78cab8c-6a3a-4b0f-ba86-44214a5aa0a3 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:05.983481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c78cab8c-6a3a-4b0f-ba86-44214a5aa0a3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2384d806893b4f22b22138880a44d4ad'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:05.997444 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c78cab8c-6a3a-4b0f-ba86-44214a5aa0a3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2384d806893b4f22b22138880a44d4ad'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:06.029368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c78cab8c-6a3a-4b0f-ba86-44214a5aa0a3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2384d806893b4f22b22138880a44d4ad'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:06.038286 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:INSERT=3,keystone:DELETE=8 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:06.057371 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c78cab8c-6a3a-4b0f-ba86-44214a5aa0a3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2384d806893b4f22b22138880a44d4ad'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:06.058757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c78cab8c-6a3a-4b0f-ba86-44214a5aa0a3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2384d806893b4f22b22138880a44d4ad'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:06.072180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1342/2686] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:05 2022] DELETE /identity/v3/users/2384d806893b4f22b22138880a44d4ad => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:06.073448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32650ae9-a069-4d4c-8141-eddec5eb371d None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:06.073738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32650ae9-a069-4d4c-8141-eddec5eb371d None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:06.074024 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32650ae9-a069-4d4c-8141-eddec5eb371d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:06.107147 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-32650ae9-a069-4d4c-8141-eddec5eb371d None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:06.109767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-32650ae9-a069-4d4c-8141-eddec5eb371d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:06.207215 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:06.231625 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1345/2687] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:06.255701 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78cab8c-6a3a-4b0f-ba86-44214a5aa0a3 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:06.258467 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c78cab8c-6a3a-4b0f-ba86-44214a5aa0a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:06.371809 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78cab8c-6a3a-4b0f-ba86-44214a5aa0a3 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:06.375917 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddb8636-ecf0-4f1b-8317-734376ed87c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:06.376696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddb8636-ecf0-4f1b-8317-734376ed87c3 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:06.377040 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddb8636-ecf0-4f1b-8317-734376ed87c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:06.377381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddb8636-ecf0-4f1b-8317-734376ed87c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:06.395487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eddb8636-ecf0-4f1b-8317-734376ed87c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:06.397698 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eddb8636-ecf0-4f1b-8317-734376ed87c3 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:06.416156 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1343/2688] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:06.505919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32650ae9-a069-4d4c-8141-eddec5eb371d None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:06.505919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-32650ae9-a069-4d4c-8141-eddec5eb371d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:06.521985 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32650ae9-a069-4d4c-8141-eddec5eb371d None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:06.528787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e1b6f3-c0ae-4068-9a50-6b9226fb4737 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:06.530559 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e1b6f3-c0ae-4068-9a50-6b9226fb4737 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:06.530825 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e1b6f3-c0ae-4068-9a50-6b9226fb4737 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:06.531559 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e1b6f3-c0ae-4068-9a50-6b9226fb4737 admin admin] PATH_INFO: `/v3/users/c5b38375d6ff4d0db0f3f7bc89ca8798` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:06.532888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e1b6f3-c0ae-4068-9a50-6b9226fb4737 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c5b38375d6ff4d0db0f3f7bc89ca8798)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:06.534919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e1b6f3-c0ae-4068-9a50-6b9226fb4737 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:06.588575 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-90e1b6f3-c0ae-4068-9a50-6b9226fb4737 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c5b38375d6ff4d0db0f3f7bc89ca8798'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:06.589455 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-90e1b6f3-c0ae-4068-9a50-6b9226fb4737 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c5b38375d6ff4d0db0f3f7bc89ca8798'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:06.590097 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-90e1b6f3-c0ae-4068-9a50-6b9226fb4737 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c5b38375d6ff4d0db0f3f7bc89ca8798'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:06.598947 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-90e1b6f3-c0ae-4068-9a50-6b9226fb4737 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c5b38375d6ff4d0db0f3f7bc89ca8798'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:06.613528 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-90e1b6f3-c0ae-4068-9a50-6b9226fb4737 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c5b38375d6ff4d0db0f3f7bc89ca8798'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:06.628360 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1346/2689] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:06 2022] DELETE /identity/v3/users/c5b38375d6ff4d0db0f3f7bc89ca8798 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:06.922917 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddb8636-ecf0-4f1b-8317-734376ed87c3 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:06.936367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddb8636-ecf0-4f1b-8317-734376ed87c3 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:07.020955 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb53331-8cb0-4971-87f4-9159624c7e2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:07.021625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb53331-8cb0-4971-87f4-9159624c7e2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:07.022012 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb53331-8cb0-4971-87f4-9159624c7e2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:07.022297 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb53331-8cb0-4971-87f4-9159624c7e2f admin admin] PATH_INFO: `/v3/projects/64993786e21b45beb7829ceaede4777a` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:07.023460 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb53331-8cb0-4971-87f4-9159624c7e2f admin admin] RBAC: Authorizing `identity:delete_project(project_id=64993786e21b45beb7829ceaede4777a)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:07.025465 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb53331-8cb0-4971-87f4-9159624c7e2f admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:07.077434 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-7bb53331-8cb0-4971-87f4-9159624c7e2f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '64993786e21b45beb7829ceaede4777a', 'request_id': 'req-7bb53331-8cb0-4971-87f4-9159624c7e2f'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:07.091345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-7bb53331-8cb0-4971-87f4-9159624c7e2f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '64993786e21b45beb7829ceaede4777a', 'request_id': 'req-7bb53331-8cb0-4971-87f4-9159624c7e2f'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:07.097844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-7bb53331-8cb0-4971-87f4-9159624c7e2f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '64993786e21b45beb7829ceaede4777a', 'request_id': 'req-7bb53331-8cb0-4971-87f4-9159624c7e2f'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:07.111901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-7bb53331-8cb0-4971-87f4-9159624c7e2f admin admin] The token cache is being invalidate because project 64993786e21b45beb7829ceaede4777a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:07.112196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-7bb53331-8cb0-4971-87f4-9159624c7e2f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:07.112993 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-7bb53331-8cb0-4971-87f4-9159624c7e2f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:07.114717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1344/2690] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:06 2022] DELETE /identity/v3/projects/64993786e21b45beb7829ceaede4777a => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:07.120069 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a265ee17-2301-4311-8129-5bc7c1a26530 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:07.120696 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a265ee17-2301-4311-8129-5bc7c1a26530 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:07.121062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a265ee17-2301-4311-8129-5bc7c1a26530 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:07.141870 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-a265ee17-2301-4311-8129-5bc7c1a26530 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:07.143226 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-a265ee17-2301-4311-8129-5bc7c1a26530 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:07.207278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1347/2691] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:07.216967 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5605c7c-4e32-4c3b-b308-5c01e00bf7c1 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:07.217255 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5605c7c-4e32-4c3b-b308-5c01e00bf7c1 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:07.217532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5605c7c-4e32-4c3b-b308-5c01e00bf7c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:07.238323 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-f5605c7c-4e32-4c3b-b308-5c01e00bf7c1 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:07.241844 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f5605c7c-4e32-4c3b-b308-5c01e00bf7c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:07.250520 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1345/2692] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:07.255004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a265ee17-2301-4311-8129-5bc7c1a26530 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:07.268109 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a265ee17-2301-4311-8129-5bc7c1a26530 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:07.274636 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e16713-8bf5-4006-a058-927334ee2330 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:07.275400 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e16713-8bf5-4006-a058-927334ee2330 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:07.275782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e16713-8bf5-4006-a058-927334ee2330 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:07.276153 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e16713-8bf5-4006-a058-927334ee2330 admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:07.276931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e16713-8bf5-4006-a058-927334ee2330 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:38:07.277597 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e16713-8bf5-4006-a058-927334ee2330 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:07.279390 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e16713-8bf5-4006-a058-927334ee2330 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:07.297738 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1348/2693] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:38:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:38:07.300484 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5605c7c-4e32-4c3b-b308-5c01e00bf7c1 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:07.310373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5605c7c-4e32-4c3b-b308-5c01e00bf7c1 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:07.316557 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b85653a-d5aa-4062-8f08-c741ce6de48b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:07.317368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b85653a-d5aa-4062-8f08-c741ce6de48b admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:07.317778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b85653a-d5aa-4062-8f08-c741ce6de48b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:07.318146 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b85653a-d5aa-4062-8f08-c741ce6de48b admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:07.318807 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e16713-8bf5-4006-a058-927334ee2330 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:07.319010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b85653a-d5aa-4062-8f08-c741ce6de48b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:07.320695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-69e16713-8bf5-4006-a058-927334ee2330 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:07.321821 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b85653a-d5aa-4062-8f08-c741ce6de48b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:07.332254 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:07.336378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e16713-8bf5-4006-a058-927334ee2330 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:07.358153 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81952558-9810-40ff-8877-f1341ef6876f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:07.359011 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81952558-9810-40ff-8877-f1341ef6876f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:07.359298 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81952558-9810-40ff-8877-f1341ef6876f admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:07.359715 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81952558-9810-40ff-8877-f1341ef6876f admin admin] PATH_INFO: `/v3/projects/ac15ca31cae8480ca7680a077033e129` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:07.360169 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1346/2694] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:07 2022] POST /identity/v3/projects => generated 365 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:07.361079 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81952558-9810-40ff-8877-f1341ef6876f admin admin] RBAC: Authorizing `identity:delete_project(project_id=ac15ca31cae8480ca7680a077033e129)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:07.363133 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81952558-9810-40ff-8877-f1341ef6876f admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:07.367718 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b85653a-d5aa-4062-8f08-c741ce6de48b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:07.446296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:07.452847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-81952558-9810-40ff-8877-f1341ef6876f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ac15ca31cae8480ca7680a077033e129', 'request_id': 'req-81952558-9810-40ff-8877-f1341ef6876f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:07.457997 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b85653a-d5aa-4062-8f08-c741ce6de48b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:07.465463 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-81952558-9810-40ff-8877-f1341ef6876f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ac15ca31cae8480ca7680a077033e129', 'request_id': 'req-81952558-9810-40ff-8877-f1341ef6876f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:07.472477 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-81952558-9810-40ff-8877-f1341ef6876f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ac15ca31cae8480ca7680a077033e129', 'request_id': 'req-81952558-9810-40ff-8877-f1341ef6876f'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:07.488354 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-81952558-9810-40ff-8877-f1341ef6876f admin admin] The token cache is being invalidate because project ac15ca31cae8480ca7680a077033e129 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:07.488742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-81952558-9810-40ff-8877-f1341ef6876f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:07.490761 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-81952558-9810-40ff-8877-f1341ef6876f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:07.492587 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1349/2695] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:07 2022] DELETE /identity/v3/projects/ac15ca31cae8480ca7680a077033e129 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:38:07.493131 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81952558-9810-40ff-8877-f1341ef6876f admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:07.494447 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-81952558-9810-40ff-8877-f1341ef6876f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:07.520711 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77ecb1f-2268-4293-a92d-9581984727ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:07.521646 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77ecb1f-2268-4293-a92d-9581984727ba admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:07.522020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77ecb1f-2268-4293-a92d-9581984727ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:07.522389 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77ecb1f-2268-4293-a92d-9581984727ba admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:07.523134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77ecb1f-2268-4293-a92d-9581984727ba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:07.525353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77ecb1f-2268-4293-a92d-9581984727ba admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:07.555050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1347/2696] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:07 2022] POST /identity/v3/users => generated 283 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:38:07.555050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77ecb1f-2268-4293-a92d-9581984727ba admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:07.557163 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d77ecb1f-2268-4293-a92d-9581984727ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:07.566584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81952558-9810-40ff-8877-f1341ef6876f admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:07.570675 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77ecb1f-2268-4293-a92d-9581984727ba admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:07.571472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7443ab2b-e2a4-422f-a208-2332b2d9fc64 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:07.572095 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7443ab2b-e2a4-422f-a208-2332b2d9fc64 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:07.572596 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7443ab2b-e2a4-422f-a208-2332b2d9fc64 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:07.572874 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7443ab2b-e2a4-422f-a208-2332b2d9fc64 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:07.575901 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7443ab2b-e2a4-422f-a208-2332b2d9fc64 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:07.576134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde1b96e-f87e-4d67-a8ec-1134eaa9e9c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:07.576889 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde1b96e-f87e-4d67-a8ec-1134eaa9e9c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:07.577264 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde1b96e-f87e-4d67-a8ec-1134eaa9e9c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:07.577635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde1b96e-f87e-4d67-a8ec-1134eaa9e9c1 admin admin] PATH_INFO: `/v3/users/57d4996e18674471a33c4486bf6bf4b8` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:07.578857 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde1b96e-f87e-4d67-a8ec-1134eaa9e9c1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=57d4996e18674471a33c4486bf6bf4b8)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:07.580939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde1b96e-f87e-4d67-a8ec-1134eaa9e9c1 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:07.636055 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cde1b96e-f87e-4d67-a8ec-1134eaa9e9c1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '57d4996e18674471a33c4486bf6bf4b8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:07.645362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cde1b96e-f87e-4d67-a8ec-1134eaa9e9c1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '57d4996e18674471a33c4486bf6bf4b8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:07.655034 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cde1b96e-f87e-4d67-a8ec-1134eaa9e9c1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '57d4996e18674471a33c4486bf6bf4b8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:07.659727 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7443ab2b-e2a4-422f-a208-2332b2d9fc64 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:07.661930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7443ab2b-e2a4-422f-a208-2332b2d9fc64 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:07.672349 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cde1b96e-f87e-4d67-a8ec-1134eaa9e9c1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '57d4996e18674471a33c4486bf6bf4b8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:07.672607 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cde1b96e-f87e-4d67-a8ec-1134eaa9e9c1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '57d4996e18674471a33c4486bf6bf4b8'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:07.674242 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:07.678692 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1350/2697] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:07 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:07.679398 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7443ab2b-e2a4-422f-a208-2332b2d9fc64 service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:07.681053 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7443ab2b-e2a4-422f-a208-2332b2d9fc64 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:07.682903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1348/2698] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:07 2022] DELETE /identity/v3/users/57d4996e18674471a33c4486bf6bf4b8 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:07.704742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7443ab2b-e2a4-422f-a208-2332b2d9fc64 service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:07.792895 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebac8e01-8aac-405d-a022-0836de3c6884 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:07.794586 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebac8e01-8aac-405d-a022-0836de3c6884 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:07.795103 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebac8e01-8aac-405d-a022-0836de3c6884 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:07.795834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebac8e01-8aac-405d-a022-0836de3c6884 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:07.796659 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebac8e01-8aac-405d-a022-0836de3c6884 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:07.798415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebac8e01-8aac-405d-a022-0836de3c6884 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:07.814412 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1351/2699] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:07 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:07.819192 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde1b96e-f87e-4d67-a8ec-1134eaa9e9c1 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:07.821219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-cde1b96e-f87e-4d67-a8ec-1134eaa9e9c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:07.838809 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde1b96e-f87e-4d67-a8ec-1134eaa9e9c1 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:07.844749 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72966871-c0e9-42f4-98ba-77c081b937ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:07.845400 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72966871-c0e9-42f4-98ba-77c081b937ca admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:07.845755 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72966871-c0e9-42f4-98ba-77c081b937ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:07.846019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72966871-c0e9-42f4-98ba-77c081b937ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:07.847483 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72966871-c0e9-42f4-98ba-77c081b937ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:07.850847 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72966871-c0e9-42f4-98ba-77c081b937ca admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:07.866609 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1349/2700] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:07 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:38:07.871763 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebac8e01-8aac-405d-a022-0836de3c6884 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:07.883965 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebac8e01-8aac-405d-a022-0836de3c6884 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:07.889986 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5fb034-027b-40c2-a8be-4022ccfebd65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:07.890773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5fb034-027b-40c2-a8be-4022ccfebd65 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:07.892087 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5fb034-027b-40c2-a8be-4022ccfebd65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:07.892087 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5fb034-027b-40c2-a8be-4022ccfebd65 admin admin] PATH_INFO: `/v3/projects/d998f72e993c41068c7e07572ce4be4a/users/df601a593b0a4976844f40ec660a5871/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:07.911472 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5fb034-027b-40c2-a8be-4022ccfebd65 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d998f72e993c41068c7e07572ce4be4a, user_id=df601a593b0a4976844f40ec660a5871, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:07.915351 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5fb034-027b-40c2-a8be-4022ccfebd65 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:07.941627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=95,keystone:DELETE=3,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:07.955780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1352/2701] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:07 2022] PUT /identity/v3/projects/d998f72e993c41068c7e07572ce4be4a/users/df601a593b0a4976844f40ec660a5871/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:07.969334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d038d5c-73cb-4956-be2b-e5048d18aca1 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:07.969883 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d038d5c-73cb-4956-be2b-e5048d18aca1 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:07.970564 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d038d5c-73cb-4956-be2b-e5048d18aca1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:08.003756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-7d038d5c-73cb-4956-be2b-e5048d18aca1 None None] MFA Rules not processed for user `df601a593b0a4976844f40ec660a5871`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:08.005770 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7d038d5c-73cb-4956-be2b-e5048d18aca1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:08.085207 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1350/2702] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:07 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:08.103205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5fb034-027b-40c2-a8be-4022ccfebd65 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:08.105406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-cd5fb034-027b-40c2-a8be-4022ccfebd65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:08.117339 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d038d5c-73cb-4956-be2b-e5048d18aca1 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:08.118221 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-7d038d5c-73cb-4956-be2b-e5048d18aca1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:08.201324 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d038d5c-73cb-4956-be2b-e5048d18aca1 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:08.205340 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5fb034-027b-40c2-a8be-4022ccfebd65 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:08.207117 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b5a7e9-781b-447f-ae71-d4b998fbdf6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:08.207968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b5a7e9-781b-447f-ae71-d4b998fbdf6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:08.209012 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b5a7e9-781b-447f-ae71-d4b998fbdf6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:08.209295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b5a7e9-781b-447f-ae71-d4b998fbdf6e admin admin] PATH_INFO: `/v3/projects/a95fe3175a484fd8a3d91f6564460da1` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:08.210740 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b5a7e9-781b-447f-ae71-d4b998fbdf6e admin admin] RBAC: Authorizing `identity:delete_project(project_id=a95fe3175a484fd8a3d91f6564460da1)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:08.211393 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24d749b-88bd-443e-bb0b-344e87eff738 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:08.212072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24d749b-88bd-443e-bb0b-344e87eff738 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:08.213706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24d749b-88bd-443e-bb0b-344e87eff738 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:08.213706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24d749b-88bd-443e-bb0b-344e87eff738 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:08.213944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b5a7e9-781b-447f-ae71-d4b998fbdf6e admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:08.226839 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24d749b-88bd-443e-bb0b-344e87eff738 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:08.228687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24d749b-88bd-443e-bb0b-344e87eff738 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:08.238527 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1353/2703] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:08 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:08.282345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-90b5a7e9-781b-447f-ae71-d4b998fbdf6e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a95fe3175a484fd8a3d91f6564460da1', 'request_id': 'req-90b5a7e9-781b-447f-ae71-d4b998fbdf6e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:08.292551 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-90b5a7e9-781b-447f-ae71-d4b998fbdf6e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a95fe3175a484fd8a3d91f6564460da1', 'request_id': 'req-90b5a7e9-781b-447f-ae71-d4b998fbdf6e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:08.302954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-90b5a7e9-781b-447f-ae71-d4b998fbdf6e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a95fe3175a484fd8a3d91f6564460da1', 'request_id': 'req-90b5a7e9-781b-447f-ae71-d4b998fbdf6e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:08.320067 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-90b5a7e9-781b-447f-ae71-d4b998fbdf6e admin admin] The token cache is being invalidate because project a95fe3175a484fd8a3d91f6564460da1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:08.320445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-90b5a7e9-781b-447f-ae71-d4b998fbdf6e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:08.322499 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-90b5a7e9-781b-447f-ae71-d4b998fbdf6e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:08.324355 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1351/2704] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:08 2022] DELETE /identity/v3/projects/a95fe3175a484fd8a3d91f6564460da1 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:08.328624 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30416089-318e-46a5-a41e-e3b5ab1357a7 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:08.329359 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30416089-318e-46a5-a41e-e3b5ab1357a7 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:08.329359 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30416089-318e-46a5-a41e-e3b5ab1357a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:08.351523 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-30416089-318e-46a5-a41e-e3b5ab1357a7 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:08.352930 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-30416089-318e-46a5-a41e-e3b5ab1357a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:08.431963 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1354/2705] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:08.440009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b0de4a-2fc7-48f3-b3f4-03ed579329b1 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:08.440853 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b0de4a-2fc7-48f3-b3f4-03ed579329b1 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:08.441235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b0de4a-2fc7-48f3-b3f4-03ed579329b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:08.461527 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:08.469054 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-47b0de4a-2fc7-48f3-b3f4-03ed579329b1 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:08.470935 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-47b0de4a-2fc7-48f3-b3f4-03ed579329b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:08.480735 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1352/2706] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:08.484519 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30416089-318e-46a5-a41e-e3b5ab1357a7 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:08.495607 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30416089-318e-46a5-a41e-e3b5ab1357a7 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:08.500640 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a718378-699d-4563-8817-e66795f86b26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:08.501572 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a718378-699d-4563-8817-e66795f86b26 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:08.502009 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a718378-699d-4563-8817-e66795f86b26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:08.502301 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a718378-699d-4563-8817-e66795f86b26 admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:08.502950 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a718378-699d-4563-8817-e66795f86b26 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:38:08.503309 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a718378-699d-4563-8817-e66795f86b26 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:08.504965 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a718378-699d-4563-8817-e66795f86b26 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:08.521888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1355/2707] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:38:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:38:08.526665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b0de4a-2fc7-48f3-b3f4-03ed579329b1 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:08.547340 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b0de4a-2fc7-48f3-b3f4-03ed579329b1 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:08.558664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cd87da-fa57-439f-8673-d4c5e7609fb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:08.559328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cd87da-fa57-439f-8673-d4c5e7609fb4 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:08.559597 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cd87da-fa57-439f-8673-d4c5e7609fb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:08.559871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cd87da-fa57-439f-8673-d4c5e7609fb4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:08.560548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58cd87da-fa57-439f-8673-d4c5e7609fb4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:08.566668 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58cd87da-fa57-439f-8673-d4c5e7609fb4 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:08.612429 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1353/2708] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:08 2022] POST /identity/v3/projects => generated 381 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:08.615091 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a718378-699d-4563-8817-e66795f86b26 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:08.640082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a718378-699d-4563-8817-e66795f86b26 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:08.710495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347f9960-7c8e-4d14-a969-a3709dee5863 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:08.711195 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347f9960-7c8e-4d14-a969-a3709dee5863 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:08.711548 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347f9960-7c8e-4d14-a969-a3709dee5863 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:08.711894 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347f9960-7c8e-4d14-a969-a3709dee5863 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:08.712643 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347f9960-7c8e-4d14-a969-a3709dee5863 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:08.714696 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347f9960-7c8e-4d14-a969-a3709dee5863 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:08.736723 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1356/2709] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:08 2022] POST /identity/v3/users => generated 291 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:08.741799 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cd87da-fa57-439f-8673-d4c5e7609fb4 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:08.759581 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cd87da-fa57-439f-8673-d4c5e7609fb4 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:08.764919 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bde7967-0d43-4248-be8a-46af629e1d5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:08.765418 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bde7967-0d43-4248-be8a-46af629e1d5e admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:08.765759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bde7967-0d43-4248-be8a-46af629e1d5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:08.766091 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bde7967-0d43-4248-be8a-46af629e1d5e admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:08.766801 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bde7967-0d43-4248-be8a-46af629e1d5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:08.768660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bde7967-0d43-4248-be8a-46af629e1d5e admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:08.780940 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1354/2710] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:08 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:08.785071 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347f9960-7c8e-4d14-a969-a3709dee5863 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:08.796224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347f9960-7c8e-4d14-a969-a3709dee5863 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:08.801582 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7527b93-eed0-47a4-bde6-9ccbe67bd201 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:08.802772 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7527b93-eed0-47a4-bde6-9ccbe67bd201 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:08.802772 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7527b93-eed0-47a4-bde6-9ccbe67bd201 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:08.803001 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7527b93-eed0-47a4-bde6-9ccbe67bd201 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:08.803608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7527b93-eed0-47a4-bde6-9ccbe67bd201 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:08.805261 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7527b93-eed0-47a4-bde6-9ccbe67bd201 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:08.816480 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1357/2711] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:08.821110 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bde7967-0d43-4248-be8a-46af629e1d5e admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:08.831654 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bde7967-0d43-4248-be8a-46af629e1d5e admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:08.836841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd5fd2c-8543-4eea-b2c3-2898e43f941c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:08.837566 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd5fd2c-8543-4eea-b2c3-2898e43f941c admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:08.837840 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd5fd2c-8543-4eea-b2c3-2898e43f941c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:08.838110 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd5fd2c-8543-4eea-b2c3-2898e43f941c admin admin] PATH_INFO: `/v3/projects/9e4513b40ae04db0812c7743db67d245/users/b634f2351a0d402188959eec210b9ded/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:08.855061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd5fd2c-8543-4eea-b2c3-2898e43f941c admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e4513b40ae04db0812c7743db67d245, user_id=b634f2351a0d402188959eec210b9ded, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:08.857961 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd5fd2c-8543-4eea-b2c3-2898e43f941c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:08.873366 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1355/2712] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:08 2022] PUT /identity/v3/projects/9e4513b40ae04db0812c7743db67d245/users/b634f2351a0d402188959eec210b9ded/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:38:08.884476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2439b53-9573-4b0d-baf7-8221c9b639b2 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:08.884764 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2439b53-9573-4b0d-baf7-8221c9b639b2 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:08.885037 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2439b53-9573-4b0d-baf7-8221c9b639b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:08.898816 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:08.915543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-b2439b53-9573-4b0d-baf7-8221c9b639b2 None None] MFA Rules not processed for user `b634f2351a0d402188959eec210b9ded`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:08.916926 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-b2439b53-9573-4b0d-baf7-8221c9b639b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:08.990558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1358/2713] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:08 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:08.995763 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd5fd2c-8543-4eea-b2c3-2898e43f941c admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:09.073483 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd5fd2c-8543-4eea-b2c3-2898e43f941c admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:09.078320 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a1faf5-57b1-4a27-a765-57e7f6cfa62f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:09.078939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a1faf5-57b1-4a27-a765-57e7f6cfa62f admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:09.079216 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a1faf5-57b1-4a27-a765-57e7f6cfa62f admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:09.079490 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a1faf5-57b1-4a27-a765-57e7f6cfa62f admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:09.080119 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a1faf5-57b1-4a27-a765-57e7f6cfa62f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:09.082111 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a1faf5-57b1-4a27-a765-57e7f6cfa62f admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:09.110534 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1356/2714] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:08 2022] POST /identity/v3/projects => generated 385 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:09.114840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2439b53-9573-4b0d-baf7-8221c9b639b2 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:09.126039 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2439b53-9573-4b0d-baf7-8221c9b639b2 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:09.185990 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725dc1eb-fe6a-4d96-aec8-e8e0a63e43fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:09.186603 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725dc1eb-fe6a-4d96-aec8-e8e0a63e43fe admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:09.186882 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725dc1eb-fe6a-4d96-aec8-e8e0a63e43fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:09.187186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725dc1eb-fe6a-4d96-aec8-e8e0a63e43fe admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:09.187808 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725dc1eb-fe6a-4d96-aec8-e8e0a63e43fe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:09.189746 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725dc1eb-fe6a-4d96-aec8-e8e0a63e43fe admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:09.221045 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1359/2715] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:09 2022] POST /identity/v3/users => generated 299 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:09.222897 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a1faf5-57b1-4a27-a765-57e7f6cfa62f admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:09.236430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a1faf5-57b1-4a27-a765-57e7f6cfa62f admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:09.241902 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0014457-83b5-40ad-8f26-37a7139d5049 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:09.242511 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0014457-83b5-40ad-8f26-37a7139d5049 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:09.242788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0014457-83b5-40ad-8f26-37a7139d5049 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:09.243058 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0014457-83b5-40ad-8f26-37a7139d5049 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:09.243731 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0014457-83b5-40ad-8f26-37a7139d5049 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:09.245361 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0014457-83b5-40ad-8f26-37a7139d5049 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:09.257046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1357/2716] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:09 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:09.262016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725dc1eb-fe6a-4d96-aec8-e8e0a63e43fe admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:09.273577 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725dc1eb-fe6a-4d96-aec8-e8e0a63e43fe admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:09.279185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acf7abc-4bd2-415b-9ed3-6e6e97d16ca1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:09.279857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acf7abc-4bd2-415b-9ed3-6e6e97d16ca1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:09.280139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acf7abc-4bd2-415b-9ed3-6e6e97d16ca1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:09.280456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acf7abc-4bd2-415b-9ed3-6e6e97d16ca1 admin admin] PATH_INFO: `/v3/projects/130e2b5e328f41dd8868996e124a5138/users/0f988c2ddcf248aab116d7b315ff2356/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:09.297848 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acf7abc-4bd2-415b-9ed3-6e6e97d16ca1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=130e2b5e328f41dd8868996e124a5138, user_id=0f988c2ddcf248aab116d7b315ff2356, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:09.300871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acf7abc-4bd2-415b-9ed3-6e6e97d16ca1 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:09.318998 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1360/2717] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:09 2022] PUT /identity/v3/projects/130e2b5e328f41dd8868996e124a5138/users/0f988c2ddcf248aab116d7b315ff2356/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:09.323142 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0014457-83b5-40ad-8f26-37a7139d5049 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:09.333817 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0014457-83b5-40ad-8f26-37a7139d5049 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:09.394352 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c8de8c-6855-484e-8679-91f9c29968e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:09.395053 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c8de8c-6855-484e-8679-91f9c29968e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:09.395338 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c8de8c-6855-484e-8679-91f9c29968e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:09.395645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c8de8c-6855-484e-8679-91f9c29968e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:09.396348 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c8de8c-6855-484e-8679-91f9c29968e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:09.397952 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c8de8c-6855-484e-8679-91f9c29968e1 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:09.408553 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1358/2718] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:09 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:09.412762 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acf7abc-4bd2-415b-9ed3-6e6e97d16ca1 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:09.422958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acf7abc-4bd2-415b-9ed3-6e6e97d16ca1 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:09.428516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c0ba80-b8bc-4100-8f92-9e7c9444ecdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:09.429174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c0ba80-b8bc-4100-8f92-9e7c9444ecdd admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:09.429456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c0ba80-b8bc-4100-8f92-9e7c9444ecdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:09.429734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c0ba80-b8bc-4100-8f92-9e7c9444ecdd admin admin] PATH_INFO: `/v3/projects/130e2b5e328f41dd8868996e124a5138/users/0f988c2ddcf248aab116d7b315ff2356/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:09.431653 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c0ba80-b8bc-4100-8f92-9e7c9444ecdd admin admin] RBAC: Authorizing `identity:create_grant(project_id=130e2b5e328f41dd8868996e124a5138, user_id=0f988c2ddcf248aab116d7b315ff2356, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:09.434323 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c0ba80-b8bc-4100-8f92-9e7c9444ecdd admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:09.445772 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1361/2719] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:09 2022] PUT /identity/v3/projects/130e2b5e328f41dd8868996e124a5138/users/0f988c2ddcf248aab116d7b315ff2356/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:09.459743 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97a40e4-ec05-4bc4-8d16-51988d3fef61 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:09.460030 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97a40e4-ec05-4bc4-8d16-51988d3fef61 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:09.460333 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97a40e4-ec05-4bc4-8d16-51988d3fef61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:09.485365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:09.488486 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-f97a40e4-ec05-4bc4-8d16-51988d3fef61 None None] MFA Rules not processed for user `0f988c2ddcf248aab116d7b315ff2356`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:09.491251 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f97a40e4-ec05-4bc4-8d16-51988d3fef61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:09.573065 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1359/2720] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:09 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:38:09.584563 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c0ba80-b8bc-4100-8f92-9e7c9444ecdd admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:09.586687 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-e6c0ba80-b8bc-4100-8f92-9e7c9444ecdd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:09.655247 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:09.659151 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c0ba80-b8bc-4100-8f92-9e7c9444ecdd admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:09.667234 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55311763-f42a-43b2-a400-1e922064fe2b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:09.667885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55311763-f42a-43b2-a400-1e922064fe2b service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:09.668365 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55311763-f42a-43b2-a400-1e922064fe2b service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:09.668648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55311763-f42a-43b2-a400-1e922064fe2b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:09.743587 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55311763-f42a-43b2-a400-1e922064fe2b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:09.745095 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55311763-f42a-43b2-a400-1e922064fe2b service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:09.769092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1362/2721] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:09 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:14.283495 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97a40e4-ec05-4bc4-8d16-51988d3fef61 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:14.284910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-f97a40e4-ec05-4bc4-8d16-51988d3fef61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:14.297255 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97a40e4-ec05-4bc4-8d16-51988d3fef61 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:14.300811 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7e1360-1f73-4a77-bb1f-c0d4344e8445 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:14.301429 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7e1360-1f73-4a77-bb1f-c0d4344e8445 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:14.301699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7e1360-1f73-4a77-bb1f-c0d4344e8445 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:14.302083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7e1360-1f73-4a77-bb1f-c0d4344e8445 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:14.311067 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7e1360-1f73-4a77-bb1f-c0d4344e8445 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:14.312542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7e1360-1f73-4a77-bb1f-c0d4344e8445 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:14.322533 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1360/2722] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:14 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:17.048274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55311763-f42a-43b2-a400-1e922064fe2b service cinder] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:17.049850 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-55311763-f42a-43b2-a400-1e922064fe2b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:17.132078 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55311763-f42a-43b2-a400-1e922064fe2b service cinder] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:17.137065 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301b81ba-4136-4566-8222-cfcee2e99b56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:17.138063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301b81ba-4136-4566-8222-cfcee2e99b56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:17.138554 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301b81ba-4136-4566-8222-cfcee2e99b56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:17.138834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301b81ba-4136-4566-8222-cfcee2e99b56 admin admin] PATH_INFO: `/v3/users/df601a593b0a4976844f40ec660a5871` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:17.139983 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301b81ba-4136-4566-8222-cfcee2e99b56 admin admin] RBAC: Authorizing `identity:delete_user(user_id=df601a593b0a4976844f40ec660a5871)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:17.141942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301b81ba-4136-4566-8222-cfcee2e99b56 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:17.193383 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-301b81ba-4136-4566-8222-cfcee2e99b56 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df601a593b0a4976844f40ec660a5871'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:17.194135 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-301b81ba-4136-4566-8222-cfcee2e99b56 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df601a593b0a4976844f40ec660a5871'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:17.194771 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-301b81ba-4136-4566-8222-cfcee2e99b56 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df601a593b0a4976844f40ec660a5871'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:17.202702 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-301b81ba-4136-4566-8222-cfcee2e99b56 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df601a593b0a4976844f40ec660a5871'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:17.215769 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-301b81ba-4136-4566-8222-cfcee2e99b56 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df601a593b0a4976844f40ec660a5871'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:17.229305 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1363/2723] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:17 2022] DELETE /identity/v3/users/df601a593b0a4976844f40ec660a5871 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:17.233703 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50876d-c3f7-4107-8c0c-d739d68dabfe None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:17.234206 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50876d-c3f7-4107-8c0c-d739d68dabfe None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:17.234742 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50876d-c3f7-4107-8c0c-d739d68dabfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:17.254528 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-3e50876d-c3f7-4107-8c0c-d739d68dabfe None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:17.255883 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-3e50876d-c3f7-4107-8c0c-d739d68dabfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:17.323344 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1361/2724] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:17.330462 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301b81ba-4136-4566-8222-cfcee2e99b56 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:17.331960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-301b81ba-4136-4566-8222-cfcee2e99b56 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:17.367101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:17.408410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301b81ba-4136-4566-8222-cfcee2e99b56 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:17.414897 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1acbb40-29e8-4156-a604-6132fdef70dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:17.415558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1acbb40-29e8-4156-a604-6132fdef70dd service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:17.416331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1acbb40-29e8-4156-a604-6132fdef70dd service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:17.416331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1acbb40-29e8-4156-a604-6132fdef70dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:17.431218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1acbb40-29e8-4156-a604-6132fdef70dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:17.432736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1acbb40-29e8-4156-a604-6132fdef70dd service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:17.446889 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1364/2725] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:17.802833 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50876d-c3f7-4107-8c0c-d739d68dabfe None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:17.805397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-3e50876d-c3f7-4107-8c0c-d739d68dabfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:17.837653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50876d-c3f7-4107-8c0c-d739d68dabfe None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:17.843513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ce5bf1-7861-411b-baf6-6dce1c5f6ba4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:17.844628 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ce5bf1-7861-411b-baf6-6dce1c5f6ba4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:17.844991 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ce5bf1-7861-411b-baf6-6dce1c5f6ba4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:17.845330 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ce5bf1-7861-411b-baf6-6dce1c5f6ba4 admin admin] PATH_INFO: `/v3/projects/d998f72e993c41068c7e07572ce4be4a` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:17.846603 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ce5bf1-7861-411b-baf6-6dce1c5f6ba4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d998f72e993c41068c7e07572ce4be4a)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:17.849749 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ce5bf1-7861-411b-baf6-6dce1c5f6ba4 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:17.914602 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-a7ce5bf1-7861-411b-baf6-6dce1c5f6ba4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd998f72e993c41068c7e07572ce4be4a', 'request_id': 'req-a7ce5bf1-7861-411b-baf6-6dce1c5f6ba4'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:17.922899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-a7ce5bf1-7861-411b-baf6-6dce1c5f6ba4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd998f72e993c41068c7e07572ce4be4a', 'request_id': 'req-a7ce5bf1-7861-411b-baf6-6dce1c5f6ba4'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:17.932098 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-a7ce5bf1-7861-411b-baf6-6dce1c5f6ba4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd998f72e993c41068c7e07572ce4be4a', 'request_id': 'req-a7ce5bf1-7861-411b-baf6-6dce1c5f6ba4'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:17.949571 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-a7ce5bf1-7861-411b-baf6-6dce1c5f6ba4 admin admin] The token cache is being invalidate because project d998f72e993c41068c7e07572ce4be4a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:17.950011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-a7ce5bf1-7861-411b-baf6-6dce1c5f6ba4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:17.950827 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-a7ce5bf1-7861-411b-baf6-6dce1c5f6ba4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:17.953552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1362/2726] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:17 2022] DELETE /identity/v3/projects/d998f72e993c41068c7e07572ce4be4a => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:17.958700 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0111ee52-a4cc-4a4d-a425-1833b1fcb7f5 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:17.958956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0111ee52-a4cc-4a4d-a425-1833b1fcb7f5 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:17.959234 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0111ee52-a4cc-4a4d-a425-1833b1fcb7f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:17.979908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-0111ee52-a4cc-4a4d-a425-1833b1fcb7f5 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:17.981180 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0111ee52-a4cc-4a4d-a425-1833b1fcb7f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:18.042018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1365/2727] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:18.050396 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabaac96-0ba2-49a9-916d-392ee76883dd None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:18.050709 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabaac96-0ba2-49a9-916d-392ee76883dd None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:18.050990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabaac96-0ba2-49a9-916d-392ee76883dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:18.071757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-dabaac96-0ba2-49a9-916d-392ee76883dd None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:18.073006 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-dabaac96-0ba2-49a9-916d-392ee76883dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:18.081458 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1363/2728] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:18.086103 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0111ee52-a4cc-4a4d-a425-1833b1fcb7f5 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:18.097889 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0111ee52-a4cc-4a4d-a425-1833b1fcb7f5 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:18.103232 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff2a083-eda1-49c6-baac-c8b2cba2758a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:18.103844 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff2a083-eda1-49c6-baac-c8b2cba2758a admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:18.104125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff2a083-eda1-49c6-baac-c8b2cba2758a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:18.104505 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff2a083-eda1-49c6-baac-c8b2cba2758a admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:18.105157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff2a083-eda1-49c6-baac-c8b2cba2758a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:38:18.105464 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff2a083-eda1-49c6-baac-c8b2cba2758a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:18.107070 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff2a083-eda1-49c6-baac-c8b2cba2758a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:18.123591 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1366/2729] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:38:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:38:18.125226 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabaac96-0ba2-49a9-916d-392ee76883dd None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:18.138990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:18.145765 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabaac96-0ba2-49a9-916d-392ee76883dd None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:18.153395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93547094-efe1-4706-9600-e26236087b46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:18.155599 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93547094-efe1-4706-9600-e26236087b46 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:18.155933 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93547094-efe1-4706-9600-e26236087b46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:18.156173 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93547094-efe1-4706-9600-e26236087b46 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:18.156907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93547094-efe1-4706-9600-e26236087b46 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:18.159269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93547094-efe1-4706-9600-e26236087b46 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:18.189740 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1364/2730] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:18 2022] POST /identity/v3/projects => generated 365 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:38:18.194548 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff2a083-eda1-49c6-baac-c8b2cba2758a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:18.205498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff2a083-eda1-49c6-baac-c8b2cba2758a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:18.273193 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c5e543-14bd-4d46-a110-652ddb8f0b09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:18.273808 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c5e543-14bd-4d46-a110-652ddb8f0b09 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:18.274092 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c5e543-14bd-4d46-a110-652ddb8f0b09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:18.274369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c5e543-14bd-4d46-a110-652ddb8f0b09 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:18.275019 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c5e543-14bd-4d46-a110-652ddb8f0b09 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:18.277239 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c5e543-14bd-4d46-a110-652ddb8f0b09 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:18.300870 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1367/2731] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:18 2022] POST /identity/v3/users => generated 283 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:38:18.305681 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93547094-efe1-4706-9600-e26236087b46 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:18.316054 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93547094-efe1-4706-9600-e26236087b46 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:18.320848 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a32fb8-6b97-4991-8940-7434f3fbc71a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:18.321451 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a32fb8-6b97-4991-8940-7434f3fbc71a admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:18.321723 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a32fb8-6b97-4991-8940-7434f3fbc71a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:18.321986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a32fb8-6b97-4991-8940-7434f3fbc71a admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:18.322621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a32fb8-6b97-4991-8940-7434f3fbc71a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:18.324195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a32fb8-6b97-4991-8940-7434f3fbc71a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:18.335106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1365/2732] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:18 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:18.339225 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c5e543-14bd-4d46-a110-652ddb8f0b09 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:18.350350 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c5e543-14bd-4d46-a110-652ddb8f0b09 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:18.356531 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1103cb8a-2667-4496-9c2a-5c4001520d36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:18.357149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1103cb8a-2667-4496-9c2a-5c4001520d36 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:18.357435 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1103cb8a-2667-4496-9c2a-5c4001520d36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:18.357712 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1103cb8a-2667-4496-9c2a-5c4001520d36 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:18.358364 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1103cb8a-2667-4496-9c2a-5c4001520d36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:18.359973 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1103cb8a-2667-4496-9c2a-5c4001520d36 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:18.372258 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1368/2733] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:18 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:18.376542 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a32fb8-6b97-4991-8940-7434f3fbc71a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:18.387198 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a32fb8-6b97-4991-8940-7434f3fbc71a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:18.391901 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cf4201-15b4-462b-bd01-5f77450e43ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:18.392579 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cf4201-15b4-462b-bd01-5f77450e43ec admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:18.392859 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cf4201-15b4-462b-bd01-5f77450e43ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:18.393136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cf4201-15b4-462b-bd01-5f77450e43ec admin admin] PATH_INFO: `/v3/projects/2a56262a8c254417b135199177856c4b/users/c7fe42aba5034bb7aff3d27cdce0a3ab/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:18.410208 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cf4201-15b4-462b-bd01-5f77450e43ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a56262a8c254417b135199177856c4b, user_id=c7fe42aba5034bb7aff3d27cdce0a3ab, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:18.413530 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cf4201-15b4-462b-bd01-5f77450e43ec admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:18.426978 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1366/2734] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:18 2022] PUT /identity/v3/projects/2a56262a8c254417b135199177856c4b/users/c7fe42aba5034bb7aff3d27cdce0a3ab/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:18.438048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef04be5-0c47-4477-b794-41e8aa86b0c7 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:18.438365 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef04be5-0c47-4477-b794-41e8aa86b0c7 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:18.438586 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef04be5-0c47-4477-b794-41e8aa86b0c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:18.472175 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-fef04be5-0c47-4477-b794-41e8aa86b0c7 None None] MFA Rules not processed for user `c7fe42aba5034bb7aff3d27cdce0a3ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:18.473493 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-fef04be5-0c47-4477-b794-41e8aa86b0c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:18.495955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:18.551062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1369/2735] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:18 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:18.555529 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cf4201-15b4-462b-bd01-5f77450e43ec admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:18.621492 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cf4201-15b4-462b-bd01-5f77450e43ec admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:18.626318 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87888aa-8717-4d37-82d8-fe9aa60fa028 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:18.626921 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87888aa-8717-4d37-82d8-fe9aa60fa028 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:18.627214 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87888aa-8717-4d37-82d8-fe9aa60fa028 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:18.627481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87888aa-8717-4d37-82d8-fe9aa60fa028 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:18.628113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87888aa-8717-4d37-82d8-fe9aa60fa028 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:18.630637 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87888aa-8717-4d37-82d8-fe9aa60fa028 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:18.659579 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1367/2736] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:18 2022] POST /identity/v3/projects => generated 365 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:18.664765 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef04be5-0c47-4477-b794-41e8aa86b0c7 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:18.675613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef04be5-0c47-4477-b794-41e8aa86b0c7 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:18.735798 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab33e82e-9875-4ca4-b019-a3c7e33fa827 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:18.736449 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab33e82e-9875-4ca4-b019-a3c7e33fa827 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:18.736744 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab33e82e-9875-4ca4-b019-a3c7e33fa827 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:18.737015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab33e82e-9875-4ca4-b019-a3c7e33fa827 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:18.737657 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab33e82e-9875-4ca4-b019-a3c7e33fa827 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:18.740261 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab33e82e-9875-4ca4-b019-a3c7e33fa827 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:18.762658 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1370/2737] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:18 2022] POST /identity/v3/users => generated 289 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:18.766664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87888aa-8717-4d37-82d8-fe9aa60fa028 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:18.777506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87888aa-8717-4d37-82d8-fe9aa60fa028 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:18.782127 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc8bf38-a44d-4cf3-b01b-9eec02d4d4c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:18.782724 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc8bf38-a44d-4cf3-b01b-9eec02d4d4c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:18.783002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc8bf38-a44d-4cf3-b01b-9eec02d4d4c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:18.783275 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc8bf38-a44d-4cf3-b01b-9eec02d4d4c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:18.783916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc8bf38-a44d-4cf3-b01b-9eec02d4d4c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:18.785546 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc8bf38-a44d-4cf3-b01b-9eec02d4d4c3 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:18.797252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1368/2738] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:18 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:18.802514 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab33e82e-9875-4ca4-b019-a3c7e33fa827 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:18.813607 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab33e82e-9875-4ca4-b019-a3c7e33fa827 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:18.818594 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6d9e70-3f0d-4dd0-9787-8515341d2e74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:18.819272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6d9e70-3f0d-4dd0-9787-8515341d2e74 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:18.819608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6d9e70-3f0d-4dd0-9787-8515341d2e74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:18.819890 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6d9e70-3f0d-4dd0-9787-8515341d2e74 admin admin] PATH_INFO: `/v3/projects/355ee8e1a03b47c5a73d65bb1eb45416/users/b4de97fc7a12441fba6ec067c62296c2/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:18.844888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6d9e70-3f0d-4dd0-9787-8515341d2e74 admin admin] RBAC: Authorizing `identity:create_grant(project_id=355ee8e1a03b47c5a73d65bb1eb45416, user_id=b4de97fc7a12441fba6ec067c62296c2, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:18.847163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6d9e70-3f0d-4dd0-9787-8515341d2e74 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:18.862195 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1371/2739] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:18 2022] PUT /identity/v3/projects/355ee8e1a03b47c5a73d65bb1eb45416/users/b4de97fc7a12441fba6ec067c62296c2/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:18.867449 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc8bf38-a44d-4cf3-b01b-9eec02d4d4c3 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:18.878115 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc8bf38-a44d-4cf3-b01b-9eec02d4d4c3 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:18.941259 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d41464-b22c-4c59-9936-8ab56442cd82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:18.941880 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d41464-b22c-4c59-9936-8ab56442cd82 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:18.942155 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d41464-b22c-4c59-9936-8ab56442cd82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:18.942431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d41464-b22c-4c59-9936-8ab56442cd82 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:18.943083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d41464-b22c-4c59-9936-8ab56442cd82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:18.944813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d41464-b22c-4c59-9936-8ab56442cd82 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:18.959612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1369/2740] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:18 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:38:18.960931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6d9e70-3f0d-4dd0-9787-8515341d2e74 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:18.971638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6d9e70-3f0d-4dd0-9787-8515341d2e74 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:18.977961 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b340ace2-0d63-4cca-9835-47bcd7310091 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:18.978624 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b340ace2-0d63-4cca-9835-47bcd7310091 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:18.978904 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b340ace2-0d63-4cca-9835-47bcd7310091 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:18.979183 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b340ace2-0d63-4cca-9835-47bcd7310091 admin admin] PATH_INFO: `/v3/projects/355ee8e1a03b47c5a73d65bb1eb45416/users/b4de97fc7a12441fba6ec067c62296c2/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:18.981320 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b340ace2-0d63-4cca-9835-47bcd7310091 admin admin] RBAC: Authorizing `identity:create_grant(project_id=355ee8e1a03b47c5a73d65bb1eb45416, user_id=b4de97fc7a12441fba6ec067c62296c2, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:18.983854 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b340ace2-0d63-4cca-9835-47bcd7310091 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:18.996371 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1372/2741] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:18 2022] PUT /identity/v3/projects/355ee8e1a03b47c5a73d65bb1eb45416/users/b4de97fc7a12441fba6ec067c62296c2/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:19.008783 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf48c95-fd34-440e-9757-c2a9e2f5f08e None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:19.009165 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf48c95-fd34-440e-9757-c2a9e2f5f08e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:19.009526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf48c95-fd34-440e-9757-c2a9e2f5f08e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:19.044483 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-ebf48c95-fd34-440e-9757-c2a9e2f5f08e None None] MFA Rules not processed for user `b4de97fc7a12441fba6ec067c62296c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:19.045948 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ebf48c95-fd34-440e-9757-c2a9e2f5f08e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:19.049778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:19.129660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1370/2742] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:19 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:19.141124 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b340ace2-0d63-4cca-9835-47bcd7310091 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:19.143152 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-b340ace2-0d63-4cca-9835-47bcd7310091 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:19.207165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b340ace2-0d63-4cca-9835-47bcd7310091 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:19.210958 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9116bf7d-c002-4e52-9740-caac2c0a029b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:19.211579 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9116bf7d-c002-4e52-9740-caac2c0a029b service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:19.211856 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9116bf7d-c002-4e52-9740-caac2c0a029b service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:19.212185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9116bf7d-c002-4e52-9740-caac2c0a029b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:19.222584 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9116bf7d-c002-4e52-9740-caac2c0a029b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:19.224372 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9116bf7d-c002-4e52-9740-caac2c0a029b service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:19.234321 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1373/2743] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:19 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:19.301862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf48c95-fd34-440e-9757-c2a9e2f5f08e None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:19.303610 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ebf48c95-fd34-440e-9757-c2a9e2f5f08e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:19.321534 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9116bf7d-c002-4e52-9740-caac2c0a029b service cinder] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:19.324336 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9116bf7d-c002-4e52-9740-caac2c0a029b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:19.336526 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9116bf7d-c002-4e52-9740-caac2c0a029b service cinder] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:19.343500 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8472ab-3390-40ca-8814-4bd54e0ab0da service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:19.344144 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8472ab-3390-40ca-8814-4bd54e0ab0da service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:19.344459 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8472ab-3390-40ca-8814-4bd54e0ab0da service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:19.344735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8472ab-3390-40ca-8814-4bd54e0ab0da service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:19.362032 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:19.393784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf48c95-fd34-440e-9757-c2a9e2f5f08e None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:19.399842 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dc3380-6918-46a3-8b0d-c1d13506e1e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:19.400999 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dc3380-6918-46a3-8b0d-c1d13506e1e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:19.401437 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dc3380-6918-46a3-8b0d-c1d13506e1e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:19.402134 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dc3380-6918-46a3-8b0d-c1d13506e1e2 admin admin] PATH_INFO: `/v3/users/b634f2351a0d402188959eec210b9ded` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:19.403628 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dc3380-6918-46a3-8b0d-c1d13506e1e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b634f2351a0d402188959eec210b9ded)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:19.406011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dc3380-6918-46a3-8b0d-c1d13506e1e2 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:19.421397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8472ab-3390-40ca-8814-4bd54e0ab0da service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:19.423862 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8472ab-3390-40ca-8814-4bd54e0ab0da service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:19.439904 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1374/2744] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:19 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:38:19.470926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c6dc3380-6918-46a3-8b0d-c1d13506e1e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b634f2351a0d402188959eec210b9ded'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:19.481666 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c6dc3380-6918-46a3-8b0d-c1d13506e1e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b634f2351a0d402188959eec210b9ded'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:19.494532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c6dc3380-6918-46a3-8b0d-c1d13506e1e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b634f2351a0d402188959eec210b9ded'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:19.513514 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c6dc3380-6918-46a3-8b0d-c1d13506e1e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b634f2351a0d402188959eec210b9ded'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:19.514257 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c6dc3380-6918-46a3-8b0d-c1d13506e1e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b634f2351a0d402188959eec210b9ded'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:19.522219 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1371/2745] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:19 2022] DELETE /identity/v3/users/b634f2351a0d402188959eec210b9ded => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:19.528713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa1895b-88ca-4743-b438-1aec95aa86e1 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:19.529331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa1895b-88ca-4743-b438-1aec95aa86e1 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:19.529672 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa1895b-88ca-4743-b438-1aec95aa86e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:19.553245 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-9aa1895b-88ca-4743-b438-1aec95aa86e1 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:19.554490 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9aa1895b-88ca-4743-b438-1aec95aa86e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:19.636589 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1375/2746] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:19.647695 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dc3380-6918-46a3-8b0d-c1d13506e1e2 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:19.649850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c6dc3380-6918-46a3-8b0d-c1d13506e1e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:19.674079 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:19.717412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dc3380-6918-46a3-8b0d-c1d13506e1e2 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:19.721121 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23ba575-6dcd-435b-af5c-759e48eaff50 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:19.721729 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23ba575-6dcd-435b-af5c-759e48eaff50 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:19.722000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23ba575-6dcd-435b-af5c-759e48eaff50 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:19.722264 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23ba575-6dcd-435b-af5c-759e48eaff50 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:19.733068 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23ba575-6dcd-435b-af5c-759e48eaff50 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:19.734506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23ba575-6dcd-435b-af5c-759e48eaff50 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:19.745697 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1372/2747] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:38:20.063357 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa1895b-88ca-4743-b438-1aec95aa86e1 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:20.064733 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9aa1895b-88ca-4743-b438-1aec95aa86e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:20.085409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa1895b-88ca-4743-b438-1aec95aa86e1 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:20.091748 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48314d12-01cb-43f3-9a24-bc4958011388 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:20.092401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48314d12-01cb-43f3-9a24-bc4958011388 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:20.092691 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48314d12-01cb-43f3-9a24-bc4958011388 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:20.092954 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48314d12-01cb-43f3-9a24-bc4958011388 admin admin] PATH_INFO: `/v3/projects/9e4513b40ae04db0812c7743db67d245` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:20.094141 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48314d12-01cb-43f3-9a24-bc4958011388 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e4513b40ae04db0812c7743db67d245)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:20.096826 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48314d12-01cb-43f3-9a24-bc4958011388 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:20.159692 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-48314d12-01cb-43f3-9a24-bc4958011388 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e4513b40ae04db0812c7743db67d245', 'request_id': 'req-48314d12-01cb-43f3-9a24-bc4958011388'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:20.168613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-48314d12-01cb-43f3-9a24-bc4958011388 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e4513b40ae04db0812c7743db67d245', 'request_id': 'req-48314d12-01cb-43f3-9a24-bc4958011388'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:20.175405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-48314d12-01cb-43f3-9a24-bc4958011388 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e4513b40ae04db0812c7743db67d245', 'request_id': 'req-48314d12-01cb-43f3-9a24-bc4958011388'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:20.193446 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-48314d12-01cb-43f3-9a24-bc4958011388 admin admin] The token cache is being invalidate because project 9e4513b40ae04db0812c7743db67d245 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:20.194018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-48314d12-01cb-43f3-9a24-bc4958011388 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:20.194731 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-48314d12-01cb-43f3-9a24-bc4958011388 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:20.199809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1376/2748] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:20 2022] DELETE /identity/v3/projects/9e4513b40ae04db0812c7743db67d245 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:20.202976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23ba575-6dcd-435b-af5c-759e48eaff50 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:20.219057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23ba575-6dcd-435b-af5c-759e48eaff50 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:20.312165 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4108771-af79-45fe-b29e-67a8e1a36c9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:20.313295 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4108771-af79-45fe-b29e-67a8e1a36c9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:20.313750 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4108771-af79-45fe-b29e-67a8e1a36c9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:20.314456 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4108771-af79-45fe-b29e-67a8e1a36c9e admin admin] PATH_INFO: `/v3/users/0f988c2ddcf248aab116d7b315ff2356` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:20.316518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4108771-af79-45fe-b29e-67a8e1a36c9e admin admin] RBAC: Authorizing `identity:delete_user(user_id=0f988c2ddcf248aab116d7b315ff2356)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:20.320841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4108771-af79-45fe-b29e-67a8e1a36c9e admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:20.397790 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e4108771-af79-45fe-b29e-67a8e1a36c9e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0f988c2ddcf248aab116d7b315ff2356'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:20.410016 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e4108771-af79-45fe-b29e-67a8e1a36c9e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0f988c2ddcf248aab116d7b315ff2356'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:20.419454 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e4108771-af79-45fe-b29e-67a8e1a36c9e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0f988c2ddcf248aab116d7b315ff2356'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:20.433950 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e4108771-af79-45fe-b29e-67a8e1a36c9e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0f988c2ddcf248aab116d7b315ff2356'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:20.434691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e4108771-af79-45fe-b29e-67a8e1a36c9e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0f988c2ddcf248aab116d7b315ff2356'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:20.444344 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1373/2749] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:20 2022] DELETE /identity/v3/users/0f988c2ddcf248aab116d7b315ff2356 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:20.513059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48314d12-01cb-43f3-9a24-bc4958011388 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:20.514730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-48314d12-01cb-43f3-9a24-bc4958011388 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:20.568887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:20.601857 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48314d12-01cb-43f3-9a24-bc4958011388 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:20.689175 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a607bf-17fd-4df4-86e1-4ed951eb01e1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:20.690347 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a607bf-17fd-4df4-86e1-4ed951eb01e1 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:20.690933 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a607bf-17fd-4df4-86e1-4ed951eb01e1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:20.691520 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a607bf-17fd-4df4-86e1-4ed951eb01e1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:20.695986 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d9a607bf-17fd-4df4-86e1-4ed951eb01e1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:20.791668 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4108771-af79-45fe-b29e-67a8e1a36c9e admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:20.793596 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-e4108771-af79-45fe-b29e-67a8e1a36c9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:20.809843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a607bf-17fd-4df4-86e1-4ed951eb01e1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:20.812121 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a607bf-17fd-4df4-86e1-4ed951eb01e1 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:20.827655 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1377/2750] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:20 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:20.872958 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:20.909019 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4108771-af79-45fe-b29e-67a8e1a36c9e admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:20.917132 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05e7bac-5618-4a2b-bf0b-459acece0857 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:20.918860 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05e7bac-5618-4a2b-bf0b-459acece0857 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:20.919596 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05e7bac-5618-4a2b-bf0b-459acece0857 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:20.920128 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05e7bac-5618-4a2b-bf0b-459acece0857 admin admin] PATH_INFO: `/v3/projects/130e2b5e328f41dd8868996e124a5138` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:20.922178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05e7bac-5618-4a2b-bf0b-459acece0857 admin admin] RBAC: Authorizing `identity:delete_project(project_id=130e2b5e328f41dd8868996e124a5138)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:20.924541 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05e7bac-5618-4a2b-bf0b-459acece0857 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:21.002547 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e05e7bac-5618-4a2b-bf0b-459acece0857 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '130e2b5e328f41dd8868996e124a5138', 'request_id': 'req-e05e7bac-5618-4a2b-bf0b-459acece0857'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:21.019233 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e05e7bac-5618-4a2b-bf0b-459acece0857 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '130e2b5e328f41dd8868996e124a5138', 'request_id': 'req-e05e7bac-5618-4a2b-bf0b-459acece0857'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:21.036378 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e05e7bac-5618-4a2b-bf0b-459acece0857 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '130e2b5e328f41dd8868996e124a5138', 'request_id': 'req-e05e7bac-5618-4a2b-bf0b-459acece0857'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:21.052278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e05e7bac-5618-4a2b-bf0b-459acece0857 admin admin] The token cache is being invalidate because project 130e2b5e328f41dd8868996e124a5138 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:21.052882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e05e7bac-5618-4a2b-bf0b-459acece0857 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:21.053754 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-e05e7bac-5618-4a2b-bf0b-459acece0857 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:21.062945 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1374/2751] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:20 2022] DELETE /identity/v3/projects/130e2b5e328f41dd8868996e124a5138 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:21.066609 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4f4d9c-24cb-4b12-b9ed-0affc193ab0a None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:21.067004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4f4d9c-24cb-4b12-b9ed-0affc193ab0a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:21.067977 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4f4d9c-24cb-4b12-b9ed-0affc193ab0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:21.095607 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-1e4f4d9c-24cb-4b12-b9ed-0affc193ab0a None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:21.096921 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-1e4f4d9c-24cb-4b12-b9ed-0affc193ab0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:21.242621 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1378/2752] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:21.247181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc24726-f8f3-4ec2-bb68-fef83b9a93cc None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:21.247181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc24726-f8f3-4ec2-bb68-fef83b9a93cc None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:21.247181 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc24726-f8f3-4ec2-bb68-fef83b9a93cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:21.291330 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-ffc24726-f8f3-4ec2-bb68-fef83b9a93cc None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:21.293693 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ffc24726-f8f3-4ec2-bb68-fef83b9a93cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:21.312869 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1375/2753] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:21.318437 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4f4d9c-24cb-4b12-b9ed-0affc193ab0a None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:21.340713 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4f4d9c-24cb-4b12-b9ed-0affc193ab0a None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:21.347399 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803f9a21-edf8-4ae5-8898-b4008468c87c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:21.349273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803f9a21-edf8-4ae5-8898-b4008468c87c admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:21.349782 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803f9a21-edf8-4ae5-8898-b4008468c87c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:21.350150 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803f9a21-edf8-4ae5-8898-b4008468c87c admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:21.351198 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803f9a21-edf8-4ae5-8898-b4008468c87c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:38:21.352272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803f9a21-edf8-4ae5-8898-b4008468c87c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:21.354711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803f9a21-edf8-4ae5-8898-b4008468c87c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:21.378949 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1379/2754] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:38:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:38:21.383360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc24726-f8f3-4ec2-bb68-fef83b9a93cc None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:21.397710 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc24726-f8f3-4ec2-bb68-fef83b9a93cc None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:21.404197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f5af9f-8fa0-4888-ac45-06189121d6a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:21.405447 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f5af9f-8fa0-4888-ac45-06189121d6a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:21.405705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f5af9f-8fa0-4888-ac45-06189121d6a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:21.406082 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f5af9f-8fa0-4888-ac45-06189121d6a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:21.407032 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f5af9f-8fa0-4888-ac45-06189121d6a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:21.409426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f5af9f-8fa0-4888-ac45-06189121d6a7 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:21.448801 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1376/2755] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:21 2022] POST /identity/v3/projects => generated 373 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:38:21.453986 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803f9a21-edf8-4ae5-8898-b4008468c87c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:21.467166 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803f9a21-edf8-4ae5-8898-b4008468c87c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:21.514207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=100 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:21.551292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0073a129-3373-46c1-a900-89543d0275d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:21.551938 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0073a129-3373-46c1-a900-89543d0275d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:21.552853 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0073a129-3373-46c1-a900-89543d0275d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:21.553913 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0073a129-3373-46c1-a900-89543d0275d0 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:21.555354 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0073a129-3373-46c1-a900-89543d0275d0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:21.557864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0073a129-3373-46c1-a900-89543d0275d0 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:21.584486 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1380/2756] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:21 2022] POST /identity/v3/users => generated 287 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:21.588676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f5af9f-8fa0-4888-ac45-06189121d6a7 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:21.601273 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f5af9f-8fa0-4888-ac45-06189121d6a7 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:21.607113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7934fe-4eb0-4326-b782-0244d9fa33f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:21.607876 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7934fe-4eb0-4326-b782-0244d9fa33f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:21.608229 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7934fe-4eb0-4326-b782-0244d9fa33f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:21.608603 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7934fe-4eb0-4326-b782-0244d9fa33f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:21.609340 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7934fe-4eb0-4326-b782-0244d9fa33f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:21.611080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7934fe-4eb0-4326-b782-0244d9fa33f9 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:21.632658 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1377/2757] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:21 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:21.634700 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0073a129-3373-46c1-a900-89543d0275d0 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:21.656750 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0073a129-3373-46c1-a900-89543d0275d0 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:21.665787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1437802c-9a6d-4c93-b38d-4832ec1b0cdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:21.666926 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1437802c-9a6d-4c93-b38d-4832ec1b0cdd admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:21.667497 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1437802c-9a6d-4c93-b38d-4832ec1b0cdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:21.668053 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1437802c-9a6d-4c93-b38d-4832ec1b0cdd admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:21.669436 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1437802c-9a6d-4c93-b38d-4832ec1b0cdd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:21.671948 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1437802c-9a6d-4c93-b38d-4832ec1b0cdd admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:21.691569 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1381/2758] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:21 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:21.693409 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7934fe-4eb0-4326-b782-0244d9fa33f9 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:21.704874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7934fe-4eb0-4326-b782-0244d9fa33f9 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:21.710840 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5831703-1240-4626-ad48-1654b9f2e0a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:21.711683 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5831703-1240-4626-ad48-1654b9f2e0a0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:21.712030 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5831703-1240-4626-ad48-1654b9f2e0a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:21.712402 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5831703-1240-4626-ad48-1654b9f2e0a0 admin admin] PATH_INFO: `/v3/projects/8ba23e74846c430a993a44811c753e62/users/24c51c97523d4aca93b45d1d6617cf2e/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:21.733785 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5831703-1240-4626-ad48-1654b9f2e0a0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ba23e74846c430a993a44811c753e62, user_id=24c51c97523d4aca93b45d1d6617cf2e, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:21.737581 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5831703-1240-4626-ad48-1654b9f2e0a0 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:21.759158 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1378/2759] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:21 2022] PUT /identity/v3/projects/8ba23e74846c430a993a44811c753e62/users/24c51c97523d4aca93b45d1d6617cf2e/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:38:21.773655 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c22595b-a721-462c-9944-170d83d89a7c None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:21.774128 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c22595b-a721-462c-9944-170d83d89a7c None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:21.774564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c22595b-a721-462c-9944-170d83d89a7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:21.814813 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-4c22595b-a721-462c-9944-170d83d89a7c None None] MFA Rules not processed for user `24c51c97523d4aca93b45d1d6617cf2e`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:21.816617 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-4c22595b-a721-462c-9944-170d83d89a7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:21.935441 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1382/2760] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:21 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:21.941357 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5831703-1240-4626-ad48-1654b9f2e0a0 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:22.019367 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5831703-1240-4626-ad48-1654b9f2e0a0 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:22.024963 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc551ed-4ab6-43ad-a0fd-7541da19dd6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:22.025584 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc551ed-4ab6-43ad-a0fd-7541da19dd6c admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:22.025864 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc551ed-4ab6-43ad-a0fd-7541da19dd6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:22.026131 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc551ed-4ab6-43ad-a0fd-7541da19dd6c admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:22.027057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc551ed-4ab6-43ad-a0fd-7541da19dd6c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:22.028986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc551ed-4ab6-43ad-a0fd-7541da19dd6c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:22.045635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:22.066574 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1379/2761] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:21 2022] POST /identity/v3/projects => generated 373 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:38:22.071743 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c22595b-a721-462c-9944-170d83d89a7c None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:22.086814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c22595b-a721-462c-9944-170d83d89a7c None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:22.167424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11f29f5-00ad-440a-8371-bdc0a3681bbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:22.167424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f29f5-00ad-440a-8371-bdc0a3681bbb admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:22.167424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f29f5-00ad-440a-8371-bdc0a3681bbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:22.167424 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f29f5-00ad-440a-8371-bdc0a3681bbb admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:22.167954 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11f29f5-00ad-440a-8371-bdc0a3681bbb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:22.169309 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11f29f5-00ad-440a-8371-bdc0a3681bbb admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:22.193481 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1383/2762] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:22 2022] POST /identity/v3/users => generated 293 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:38:22.198798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc551ed-4ab6-43ad-a0fd-7541da19dd6c admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:22.210861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc551ed-4ab6-43ad-a0fd-7541da19dd6c admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:22.217125 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ba25b6-1f80-4abc-adf5-66cf93301025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:22.218073 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ba25b6-1f80-4abc-adf5-66cf93301025 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:22.218618 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ba25b6-1f80-4abc-adf5-66cf93301025 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:22.219182 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ba25b6-1f80-4abc-adf5-66cf93301025 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:22.220375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ba25b6-1f80-4abc-adf5-66cf93301025 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:22.222509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ba25b6-1f80-4abc-adf5-66cf93301025 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:22.235139 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1380/2763] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:22 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:22.240485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11f29f5-00ad-440a-8371-bdc0a3681bbb admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:22.255286 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11f29f5-00ad-440a-8371-bdc0a3681bbb admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:22.260900 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2550ba9-8404-48c5-8849-cdaa5a0f5d83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:22.261706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2550ba9-8404-48c5-8849-cdaa5a0f5d83 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:22.262060 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2550ba9-8404-48c5-8849-cdaa5a0f5d83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:22.262410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2550ba9-8404-48c5-8849-cdaa5a0f5d83 admin admin] PATH_INFO: `/v3/projects/cf5d9b24fe7f4eeaa8465736ec73da5c/users/b77a98d6bdfb4b02bea91f7fda0bfe20/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:22.280627 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2550ba9-8404-48c5-8849-cdaa5a0f5d83 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf5d9b24fe7f4eeaa8465736ec73da5c, user_id=b77a98d6bdfb4b02bea91f7fda0bfe20, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:22.283370 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2550ba9-8404-48c5-8849-cdaa5a0f5d83 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:22.300181 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1384/2764] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:22 2022] PUT /identity/v3/projects/cf5d9b24fe7f4eeaa8465736ec73da5c/users/b77a98d6bdfb4b02bea91f7fda0bfe20/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:22.304640 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ba25b6-1f80-4abc-adf5-66cf93301025 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:22.318417 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ba25b6-1f80-4abc-adf5-66cf93301025 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:22.404820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994a6e4e-37bd-4c86-bf93-6224484f20a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:22.404820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994a6e4e-37bd-4c86-bf93-6224484f20a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:22.404820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994a6e4e-37bd-4c86-bf93-6224484f20a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:22.405731 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994a6e4e-37bd-4c86-bf93-6224484f20a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:22.406091 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994a6e4e-37bd-4c86-bf93-6224484f20a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:22.407733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994a6e4e-37bd-4c86-bf93-6224484f20a2 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:22.426235 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1381/2765] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:22 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:38:22.432034 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2550ba9-8404-48c5-8849-cdaa5a0f5d83 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:22.440846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:22.449865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2550ba9-8404-48c5-8849-cdaa5a0f5d83 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:22.475936 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74554c8a-ac34-477d-a80a-e750d5aa3cb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:22.476809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74554c8a-ac34-477d-a80a-e750d5aa3cb8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:22.477336 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74554c8a-ac34-477d-a80a-e750d5aa3cb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:22.477660 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74554c8a-ac34-477d-a80a-e750d5aa3cb8 admin admin] PATH_INFO: `/v3/projects/cf5d9b24fe7f4eeaa8465736ec73da5c/users/b77a98d6bdfb4b02bea91f7fda0bfe20/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:22.480165 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74554c8a-ac34-477d-a80a-e750d5aa3cb8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf5d9b24fe7f4eeaa8465736ec73da5c, user_id=b77a98d6bdfb4b02bea91f7fda0bfe20, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:22.483137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74554c8a-ac34-477d-a80a-e750d5aa3cb8 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:22.502132 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1385/2766] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:22 2022] PUT /identity/v3/projects/cf5d9b24fe7f4eeaa8465736ec73da5c/users/b77a98d6bdfb4b02bea91f7fda0bfe20/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:38:22.518896 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c648d6a1-e2b7-474a-b5eb-4b63007ca35e None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:22.519441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c648d6a1-e2b7-474a-b5eb-4b63007ca35e None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:22.519830 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c648d6a1-e2b7-474a-b5eb-4b63007ca35e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:22.564635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-c648d6a1-e2b7-474a-b5eb-4b63007ca35e None None] MFA Rules not processed for user `b77a98d6bdfb4b02bea91f7fda0bfe20`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:22.566238 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c648d6a1-e2b7-474a-b5eb-4b63007ca35e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:22.697822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1382/2767] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:22 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:22.708512 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74554c8a-ac34-477d-a80a-e750d5aa3cb8 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:22.710914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-74554c8a-ac34-477d-a80a-e750d5aa3cb8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:22.845342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74554c8a-ac34-477d-a80a-e750d5aa3cb8 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:22.851648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc113a9-2d1a-4db0-aaf2-fbdb62fc5c41 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:22.852772 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc113a9-2d1a-4db0-aaf2-fbdb62fc5c41 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:22.853252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc113a9-2d1a-4db0-aaf2-fbdb62fc5c41 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:22.853733 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc113a9-2d1a-4db0-aaf2-fbdb62fc5c41 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:22.970453 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc113a9-2d1a-4db0-aaf2-fbdb62fc5c41 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:22.972177 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc113a9-2d1a-4db0-aaf2-fbdb62fc5c41 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:22.986619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1386/2768] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:22 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 279 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:29.011350 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c648d6a1-e2b7-474a-b5eb-4b63007ca35e None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:29.086178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c648d6a1-e2b7-474a-b5eb-4b63007ca35e None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:29.091006 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7126b7-2138-4f22-9e7d-98be53a6fedd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:29.091625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7126b7-2138-4f22-9e7d-98be53a6fedd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:29.091902 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7126b7-2138-4f22-9e7d-98be53a6fedd admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:29.092174 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7126b7-2138-4f22-9e7d-98be53a6fedd admin admin] PATH_INFO: `/v3/users/24c51c97523d4aca93b45d1d6617cf2e` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:29.093329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7126b7-2138-4f22-9e7d-98be53a6fedd admin admin] RBAC: Authorizing `identity:delete_user(user_id=24c51c97523d4aca93b45d1d6617cf2e)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:29.095296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7126b7-2138-4f22-9e7d-98be53a6fedd admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:29.101047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:29.153325 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bb7126b7-2138-4f22-9e7d-98be53a6fedd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '24c51c97523d4aca93b45d1d6617cf2e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:29.162420 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bb7126b7-2138-4f22-9e7d-98be53a6fedd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '24c51c97523d4aca93b45d1d6617cf2e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:29.172879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bb7126b7-2138-4f22-9e7d-98be53a6fedd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '24c51c97523d4aca93b45d1d6617cf2e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:29.187080 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bb7126b7-2138-4f22-9e7d-98be53a6fedd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '24c51c97523d4aca93b45d1d6617cf2e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:29.187855 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-bb7126b7-2138-4f22-9e7d-98be53a6fedd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '24c51c97523d4aca93b45d1d6617cf2e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:29.195976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1383/2769] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:29 2022] DELETE /identity/v3/users/24c51c97523d4aca93b45d1d6617cf2e => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:29.200553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269836a5-4ffc-467f-b3a2-3495ebb0ea61 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:29.202143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269836a5-4ffc-467f-b3a2-3495ebb0ea61 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:29.202143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269836a5-4ffc-467f-b3a2-3495ebb0ea61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:29.222135 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-269836a5-4ffc-467f-b3a2-3495ebb0ea61 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:29.223448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-269836a5-4ffc-467f-b3a2-3495ebb0ea61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:29.295917 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1387/2770] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:29.307992 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7126b7-2138-4f22-9e7d-98be53a6fedd admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:29.309525 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-bb7126b7-2138-4f22-9e7d-98be53a6fedd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:29.377191 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7126b7-2138-4f22-9e7d-98be53a6fedd admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:29.384950 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48bc7cb-7738-4bb8-b0d5-bf99ea386b79 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:29.384950 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48bc7cb-7738-4bb8-b0d5-bf99ea386b79 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:29.384950 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48bc7cb-7738-4bb8-b0d5-bf99ea386b79 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:29.384950 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48bc7cb-7738-4bb8-b0d5-bf99ea386b79 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:29.395662 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48bc7cb-7738-4bb8-b0d5-bf99ea386b79 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:29.397123 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48bc7cb-7738-4bb8-b0d5-bf99ea386b79 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:29.408214 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1384/2771] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:29.704996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269836a5-4ffc-467f-b3a2-3495ebb0ea61 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:29.706577 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-269836a5-4ffc-467f-b3a2-3495ebb0ea61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:29.727657 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269836a5-4ffc-467f-b3a2-3495ebb0ea61 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:29.734595 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5753e5d7-19da-478d-bc3a-f2980909ae44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:29.735068 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5753e5d7-19da-478d-bc3a-f2980909ae44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:29.735331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5753e5d7-19da-478d-bc3a-f2980909ae44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:29.735601 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5753e5d7-19da-478d-bc3a-f2980909ae44 admin admin] PATH_INFO: `/v3/projects/8ba23e74846c430a993a44811c753e62` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:29.736793 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5753e5d7-19da-478d-bc3a-f2980909ae44 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ba23e74846c430a993a44811c753e62)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:29.738818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5753e5d7-19da-478d-bc3a-f2980909ae44 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:29.785931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:29.793535 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5753e5d7-19da-478d-bc3a-f2980909ae44 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ba23e74846c430a993a44811c753e62', 'request_id': 'req-5753e5d7-19da-478d-bc3a-f2980909ae44'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:29.800425 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5753e5d7-19da-478d-bc3a-f2980909ae44 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ba23e74846c430a993a44811c753e62', 'request_id': 'req-5753e5d7-19da-478d-bc3a-f2980909ae44'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:29.806459 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5753e5d7-19da-478d-bc3a-f2980909ae44 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ba23e74846c430a993a44811c753e62', 'request_id': 'req-5753e5d7-19da-478d-bc3a-f2980909ae44'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:29.819102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5753e5d7-19da-478d-bc3a-f2980909ae44 admin admin] The token cache is being invalidate because project 8ba23e74846c430a993a44811c753e62 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:29.819397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5753e5d7-19da-478d-bc3a-f2980909ae44 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:29.820040 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-5753e5d7-19da-478d-bc3a-f2980909ae44 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:29.821594 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1388/2772] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:29 2022] DELETE /identity/v3/projects/8ba23e74846c430a993a44811c753e62 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:29.825618 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48bc7cb-7738-4bb8-b0d5-bf99ea386b79 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:29.836111 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48bc7cb-7738-4bb8-b0d5-bf99ea386b79 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:29.895719 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4c8c24-560b-4032-a5c4-190ebbde1c91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:29.896540 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4c8c24-560b-4032-a5c4-190ebbde1c91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:29.896823 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4c8c24-560b-4032-a5c4-190ebbde1c91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:29.897120 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4c8c24-560b-4032-a5c4-190ebbde1c91 admin admin] PATH_INFO: `/v3/users/b77a98d6bdfb4b02bea91f7fda0bfe20` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:29.898282 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4c8c24-560b-4032-a5c4-190ebbde1c91 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b77a98d6bdfb4b02bea91f7fda0bfe20)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:29.900198 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4c8c24-560b-4032-a5c4-190ebbde1c91 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:29.929545 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:29.950200 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9a4c8c24-560b-4032-a5c4-190ebbde1c91 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b77a98d6bdfb4b02bea91f7fda0bfe20'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:29.957062 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9a4c8c24-560b-4032-a5c4-190ebbde1c91 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b77a98d6bdfb4b02bea91f7fda0bfe20'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:29.964843 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9a4c8c24-560b-4032-a5c4-190ebbde1c91 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b77a98d6bdfb4b02bea91f7fda0bfe20'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:29.978187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9a4c8c24-560b-4032-a5c4-190ebbde1c91 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b77a98d6bdfb4b02bea91f7fda0bfe20'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:29.978928 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9a4c8c24-560b-4032-a5c4-190ebbde1c91 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b77a98d6bdfb4b02bea91f7fda0bfe20'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:29.984978 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1385/2773] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:29 2022] DELETE /identity/v3/users/b77a98d6bdfb4b02bea91f7fda0bfe20 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:30.277197 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5753e5d7-19da-478d-bc3a-f2980909ae44 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:30.278903 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-5753e5d7-19da-478d-bc3a-f2980909ae44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:30.292853 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5753e5d7-19da-478d-bc3a-f2980909ae44 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:30.355076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ae1def-7b7d-4c45-a894-6cf8101d0dbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:30.355625 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ae1def-7b7d-4c45-a894-6cf8101d0dbe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:30.355911 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ae1def-7b7d-4c45-a894-6cf8101d0dbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:30.356185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ae1def-7b7d-4c45-a894-6cf8101d0dbe admin admin] PATH_INFO: `/v3/projects/cf5d9b24fe7f4eeaa8465736ec73da5c` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:30.357342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ae1def-7b7d-4c45-a894-6cf8101d0dbe admin admin] RBAC: Authorizing `identity:delete_project(project_id=cf5d9b24fe7f4eeaa8465736ec73da5c)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:30.359251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ae1def-7b7d-4c45-a894-6cf8101d0dbe admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:30.409395 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-85ae1def-7b7d-4c45-a894-6cf8101d0dbe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cf5d9b24fe7f4eeaa8465736ec73da5c', 'request_id': 'req-85ae1def-7b7d-4c45-a894-6cf8101d0dbe'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:30.415909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-85ae1def-7b7d-4c45-a894-6cf8101d0dbe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cf5d9b24fe7f4eeaa8465736ec73da5c', 'request_id': 'req-85ae1def-7b7d-4c45-a894-6cf8101d0dbe'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:30.421696 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-85ae1def-7b7d-4c45-a894-6cf8101d0dbe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cf5d9b24fe7f4eeaa8465736ec73da5c', 'request_id': 'req-85ae1def-7b7d-4c45-a894-6cf8101d0dbe'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:30.435455 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-85ae1def-7b7d-4c45-a894-6cf8101d0dbe admin admin] The token cache is being invalidate because project cf5d9b24fe7f4eeaa8465736ec73da5c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:30.435749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-85ae1def-7b7d-4c45-a894-6cf8101d0dbe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:30.436399 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-85ae1def-7b7d-4c45-a894-6cf8101d0dbe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:30.437865 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1389/2774] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:30 2022] DELETE /identity/v3/projects/cf5d9b24fe7f4eeaa8465736ec73da5c => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:30.443436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ab37e7-4603-4e85-9e04-ac62a79aa829 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:30.443926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ab37e7-4603-4e85-9e04-ac62a79aa829 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:30.444030 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ab37e7-4603-4e85-9e04-ac62a79aa829 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:30.463967 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-85ab37e7-4603-4e85-9e04-ac62a79aa829 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:30.465216 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-85ab37e7-4603-4e85-9e04-ac62a79aa829 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:30.526379 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1386/2775] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:30.534428 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8061524a-fa3a-4645-845c-56d0e6c883a2 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:30.534714 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8061524a-fa3a-4645-845c-56d0e6c883a2 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:30.535009 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8061524a-fa3a-4645-845c-56d0e6c883a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:30.555791 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-8061524a-fa3a-4645-845c-56d0e6c883a2 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:30.557050 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-8061524a-fa3a-4645-845c-56d0e6c883a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:30.565150 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1390/2776] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:30.569468 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ab37e7-4603-4e85-9e04-ac62a79aa829 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:30.579402 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ab37e7-4603-4e85-9e04-ac62a79aa829 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:30.584818 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64919ada-916c-4912-8d94-157735b9ceaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:30.585463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64919ada-916c-4912-8d94-157735b9ceaf admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:30.585739 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64919ada-916c-4912-8d94-157735b9ceaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:30.586010 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64919ada-916c-4912-8d94-157735b9ceaf admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:30.586653 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64919ada-916c-4912-8d94-157735b9ceaf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:38:30.586951 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64919ada-916c-4912-8d94-157735b9ceaf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:30.588581 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64919ada-916c-4912-8d94-157735b9ceaf admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:30.601186 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1387/2777] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:38:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:38:30.605743 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8061524a-fa3a-4645-845c-56d0e6c883a2 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:30.615021 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8061524a-fa3a-4645-845c-56d0e6c883a2 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:30.619726 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34376d71-4df9-4705-bff5-3e13930b7936 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:30.620331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34376d71-4df9-4705-bff5-3e13930b7936 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:30.620628 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34376d71-4df9-4705-bff5-3e13930b7936 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:30.620902 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34376d71-4df9-4705-bff5-3e13930b7936 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:30.621563 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34376d71-4df9-4705-bff5-3e13930b7936 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:30.623448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34376d71-4df9-4705-bff5-3e13930b7936 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:30.654343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1391/2778] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:30 2022] POST /identity/v3/projects => generated 377 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:30.658085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64919ada-916c-4912-8d94-157735b9ceaf admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:30.668477 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64919ada-916c-4912-8d94-157735b9ceaf admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:30.726761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0ab6a1-92e7-45b2-b1eb-5f52173fdb31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:30.727365 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0ab6a1-92e7-45b2-b1eb-5f52173fdb31 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:30.727639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0ab6a1-92e7-45b2-b1eb-5f52173fdb31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:30.727907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0ab6a1-92e7-45b2-b1eb-5f52173fdb31 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:30.728569 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0ab6a1-92e7-45b2-b1eb-5f52173fdb31 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:30.730491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0ab6a1-92e7-45b2-b1eb-5f52173fdb31 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:30.755351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1388/2779] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:30 2022] POST /identity/v3/users => generated 289 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:38:30.759748 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34376d71-4df9-4705-bff5-3e13930b7936 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:30.771241 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34376d71-4df9-4705-bff5-3e13930b7936 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:30.778303 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf31e28-3bdd-468c-b84f-caa488e5db5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:30.778965 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf31e28-3bdd-468c-b84f-caa488e5db5a admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:30.779246 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf31e28-3bdd-468c-b84f-caa488e5db5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:30.779521 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf31e28-3bdd-468c-b84f-caa488e5db5a admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:30.780192 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf31e28-3bdd-468c-b84f-caa488e5db5a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:30.781974 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf31e28-3bdd-468c-b84f-caa488e5db5a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:30.796295 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1392/2780] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:30 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:30.799381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0ab6a1-92e7-45b2-b1eb-5f52173fdb31 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:30.810203 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0ab6a1-92e7-45b2-b1eb-5f52173fdb31 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:30.814827 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f7b8d1-3869-42cd-904d-bbe0e9e0f230 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:30.815082 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf31e28-3bdd-468c-b84f-caa488e5db5a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:30.816098 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f7b8d1-3869-42cd-904d-bbe0e9e0f230 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:30.816954 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f7b8d1-3869-42cd-904d-bbe0e9e0f230 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:30.817281 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f7b8d1-3869-42cd-904d-bbe0e9e0f230 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:30.817492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-5cf31e28-3bdd-468c-b84f-caa488e5db5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:30.818009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f7b8d1-3869-42cd-904d-bbe0e9e0f230 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:30.819570 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f7b8d1-3869-42cd-904d-bbe0e9e0f230 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:30.830374 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf31e28-3bdd-468c-b84f-caa488e5db5a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:30.835061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1389/2781] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:30.836049 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f7b8d1-3869-42cd-904d-bbe0e9e0f230 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:30.836381 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3604776-fdbf-40a0-82f4-e4fb8c4cc476 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:30.837013 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3604776-fdbf-40a0-82f4-e4fb8c4cc476 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:30.837293 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3604776-fdbf-40a0-82f4-e4fb8c4cc476 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:30.837565 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3604776-fdbf-40a0-82f4-e4fb8c4cc476 admin admin] PATH_INFO: `/v3/users/c7fe42aba5034bb7aff3d27cdce0a3ab` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:30.838701 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3604776-fdbf-40a0-82f4-e4fb8c4cc476 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c7fe42aba5034bb7aff3d27cdce0a3ab)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:30.840675 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3604776-fdbf-40a0-82f4-e4fb8c4cc476 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:30.845796 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f7b8d1-3869-42cd-904d-bbe0e9e0f230 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:30.851302 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bad190b-40cb-48f1-a555-182b5a653a9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:30.851953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bad190b-40cb-48f1-a555-182b5a653a9b admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:30.852223 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bad190b-40cb-48f1-a555-182b5a653a9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:30.852691 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bad190b-40cb-48f1-a555-182b5a653a9b admin admin] PATH_INFO: `/v3/projects/a2dfed22bde5465ba28f70b2eb0293be/users/9d68420cfdc54d9ba60981894fbbf9dc/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:30.859067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:30.861251 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:30.870464 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bad190b-40cb-48f1-a555-182b5a653a9b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2dfed22bde5465ba28f70b2eb0293be, user_id=9d68420cfdc54d9ba60981894fbbf9dc, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:30.874310 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bad190b-40cb-48f1-a555-182b5a653a9b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:30.887474 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1390/2782] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:30 2022] PUT /identity/v3/projects/a2dfed22bde5465ba28f70b2eb0293be/users/9d68420cfdc54d9ba60981894fbbf9dc/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:30.895534 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c3604776-fdbf-40a0-82f4-e4fb8c4cc476 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c7fe42aba5034bb7aff3d27cdce0a3ab'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:30.896204 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c3604776-fdbf-40a0-82f4-e4fb8c4cc476 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c7fe42aba5034bb7aff3d27cdce0a3ab'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:30.896846 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c3604776-fdbf-40a0-82f4-e4fb8c4cc476 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c7fe42aba5034bb7aff3d27cdce0a3ab'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:30.899163 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5080f7fa-434a-4171-9155-ce5c165baf9d None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:30.899441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5080f7fa-434a-4171-9155-ce5c165baf9d None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:30.899729 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5080f7fa-434a-4171-9155-ce5c165baf9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:30.904718 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c3604776-fdbf-40a0-82f4-e4fb8c4cc476 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c7fe42aba5034bb7aff3d27cdce0a3ab'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:30.919004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c3604776-fdbf-40a0-82f4-e4fb8c4cc476 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c7fe42aba5034bb7aff3d27cdce0a3ab'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:30.930467 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-5080f7fa-434a-4171-9155-ce5c165baf9d None None] MFA Rules not processed for user `9d68420cfdc54d9ba60981894fbbf9dc`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:30.932213 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5080f7fa-434a-4171-9155-ce5c165baf9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:30.934288 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1393/2783] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:30 2022] DELETE /identity/v3/users/c7fe42aba5034bb7aff3d27cdce0a3ab => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:38:30.938638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7323a5a5-8e32-4f63-b03e-74430ed00ced None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:30.938988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7323a5a5-8e32-4f63-b03e-74430ed00ced None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:30.939274 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7323a5a5-8e32-4f63-b03e-74430ed00ced None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:30.958579 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-7323a5a5-8e32-4f63-b03e-74430ed00ced None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:30.959802 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7323a5a5-8e32-4f63-b03e-74430ed00ced None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:31.001467 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1391/2784] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:30 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:38:31.013278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5080f7fa-434a-4171-9155-ce5c165baf9d None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:31.014904 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5080f7fa-434a-4171-9155-ce5c165baf9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:31.032908 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1394/2785] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:38:31.043049 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7323a5a5-8e32-4f63-b03e-74430ed00ced None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:31.044512 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7323a5a5-8e32-4f63-b03e-74430ed00ced None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:31.076778 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5080f7fa-434a-4171-9155-ce5c165baf9d None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:31.080536 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda85fba-4b7f-459a-9bd0-e3517e5d2930 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:31.081147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda85fba-4b7f-459a-9bd0-e3517e5d2930 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:31.081426 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda85fba-4b7f-459a-9bd0-e3517e5d2930 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:31.081699 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda85fba-4b7f-459a-9bd0-e3517e5d2930 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:31.090946 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda85fba-4b7f-459a-9bd0-e3517e5d2930 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:31.092559 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda85fba-4b7f-459a-9bd0-e3517e5d2930 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:31.102491 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1392/2786] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:31 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:38:31.102913 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7323a5a5-8e32-4f63-b03e-74430ed00ced None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:31.106972 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9de400-a031-471b-959e-592d1baddb1d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:31.107619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9de400-a031-471b-959e-592d1baddb1d service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:31.107891 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9de400-a031-471b-959e-592d1baddb1d service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:31.108161 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9de400-a031-471b-959e-592d1baddb1d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:31.117996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9de400-a031-471b-959e-592d1baddb1d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:31.119440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9de400-a031-471b-959e-592d1baddb1d service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:31.129267 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1395/2787] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:38:31.429283 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda85fba-4b7f-459a-9bd0-e3517e5d2930 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:31.430850 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-cda85fba-4b7f-459a-9bd0-e3517e5d2930 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:31.465228 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda85fba-4b7f-459a-9bd0-e3517e5d2930 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:31.471646 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef20bb6-4329-4cd9-8eef-27f816d860aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:31.472300 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef20bb6-4329-4cd9-8eef-27f816d860aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:31.472619 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef20bb6-4329-4cd9-8eef-27f816d860aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:31.473036 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef20bb6-4329-4cd9-8eef-27f816d860aa admin admin] PATH_INFO: `/v3/projects/2a56262a8c254417b135199177856c4b` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:31.474602 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef20bb6-4329-4cd9-8eef-27f816d860aa admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a56262a8c254417b135199177856c4b)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:31.476895 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef20bb6-4329-4cd9-8eef-27f816d860aa admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:31.537395 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cef20bb6-4329-4cd9-8eef-27f816d860aa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a56262a8c254417b135199177856c4b', 'request_id': 'req-cef20bb6-4329-4cd9-8eef-27f816d860aa'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:31.544981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cef20bb6-4329-4cd9-8eef-27f816d860aa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a56262a8c254417b135199177856c4b', 'request_id': 'req-cef20bb6-4329-4cd9-8eef-27f816d860aa'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:31.550499 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cef20bb6-4329-4cd9-8eef-27f816d860aa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a56262a8c254417b135199177856c4b', 'request_id': 'req-cef20bb6-4329-4cd9-8eef-27f816d860aa'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:31.563784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cef20bb6-4329-4cd9-8eef-27f816d860aa admin admin] The token cache is being invalidate because project 2a56262a8c254417b135199177856c4b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:31.564074 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cef20bb6-4329-4cd9-8eef-27f816d860aa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:31.564810 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-cef20bb6-4329-4cd9-8eef-27f816d860aa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:31.566854 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1393/2788] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:31 2022] DELETE /identity/v3/projects/2a56262a8c254417b135199177856c4b => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:31.571495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9de400-a031-471b-959e-592d1baddb1d service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:31.581208 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9de400-a031-471b-959e-592d1baddb1d service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:31.617789 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:31.645678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f551fd4-18c6-4e1e-91f6-9a17dcc1e5de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:31.646411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f551fd4-18c6-4e1e-91f6-9a17dcc1e5de admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:31.646699 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f551fd4-18c6-4e1e-91f6-9a17dcc1e5de admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:31.646977 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f551fd4-18c6-4e1e-91f6-9a17dcc1e5de admin admin] PATH_INFO: `/v3/users/b4de97fc7a12441fba6ec067c62296c2` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:31.648186 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f551fd4-18c6-4e1e-91f6-9a17dcc1e5de admin admin] RBAC: Authorizing `identity:delete_user(user_id=b4de97fc7a12441fba6ec067c62296c2)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:31.650192 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f551fd4-18c6-4e1e-91f6-9a17dcc1e5de admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:31.701922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8f551fd4-18c6-4e1e-91f6-9a17dcc1e5de admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b4de97fc7a12441fba6ec067c62296c2'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:31.702593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8f551fd4-18c6-4e1e-91f6-9a17dcc1e5de admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b4de97fc7a12441fba6ec067c62296c2'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:31.703205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8f551fd4-18c6-4e1e-91f6-9a17dcc1e5de admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b4de97fc7a12441fba6ec067c62296c2'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:31.711850 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8f551fd4-18c6-4e1e-91f6-9a17dcc1e5de admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b4de97fc7a12441fba6ec067c62296c2'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:31.725631 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-8f551fd4-18c6-4e1e-91f6-9a17dcc1e5de admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b4de97fc7a12441fba6ec067c62296c2'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:31.740106 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1396/2789] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:31 2022] DELETE /identity/v3/users/b4de97fc7a12441fba6ec067c62296c2 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:38:32.026894 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef20bb6-4329-4cd9-8eef-27f816d860aa admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:32.028376 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-cef20bb6-4329-4cd9-8eef-27f816d860aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:32.037956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:32.045510 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef20bb6-4329-4cd9-8eef-27f816d860aa admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:32.110353 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1a1c23-b200-494a-9bae-24be1eaac1ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:32.110986 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1a1c23-b200-494a-9bae-24be1eaac1ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:32.111263 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1a1c23-b200-494a-9bae-24be1eaac1ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:32.111561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1a1c23-b200-494a-9bae-24be1eaac1ee admin admin] PATH_INFO: `/v3/projects/355ee8e1a03b47c5a73d65bb1eb45416` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:32.112798 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1a1c23-b200-494a-9bae-24be1eaac1ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=355ee8e1a03b47c5a73d65bb1eb45416)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:32.115359 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1a1c23-b200-494a-9bae-24be1eaac1ee admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:32.164103 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-2a1a1c23-b200-494a-9bae-24be1eaac1ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '355ee8e1a03b47c5a73d65bb1eb45416', 'request_id': 'req-2a1a1c23-b200-494a-9bae-24be1eaac1ee'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:32.170419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-2a1a1c23-b200-494a-9bae-24be1eaac1ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '355ee8e1a03b47c5a73d65bb1eb45416', 'request_id': 'req-2a1a1c23-b200-494a-9bae-24be1eaac1ee'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:32.176852 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-2a1a1c23-b200-494a-9bae-24be1eaac1ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '355ee8e1a03b47c5a73d65bb1eb45416', 'request_id': 'req-2a1a1c23-b200-494a-9bae-24be1eaac1ee'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:32.189339 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-2a1a1c23-b200-494a-9bae-24be1eaac1ee admin admin] The token cache is being invalidate because project 355ee8e1a03b47c5a73d65bb1eb45416 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:32.189625 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-2a1a1c23-b200-494a-9bae-24be1eaac1ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:32.190227 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-2a1a1c23-b200-494a-9bae-24be1eaac1ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:32.191830 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1394/2790] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:32 2022] DELETE /identity/v3/projects/355ee8e1a03b47c5a73d65bb1eb45416 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:32.196992 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b55a3f-8b87-490f-a4dd-f5b25c3bb1f6 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:32.197284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b55a3f-8b87-490f-a4dd-f5b25c3bb1f6 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:32.197583 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b55a3f-8b87-490f-a4dd-f5b25c3bb1f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:32.220104 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-d9b55a3f-8b87-490f-a4dd-f5b25c3bb1f6 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:32.221343 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d9b55a3f-8b87-490f-a4dd-f5b25c3bb1f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:32.284794 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1397/2791] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:32.293213 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45b6129-b421-4f26-8815-08e38e4e0204 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:32.293486 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45b6129-b421-4f26-8815-08e38e4e0204 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:32.293759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45b6129-b421-4f26-8815-08e38e4e0204 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:32.314404 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-e45b6129-b421-4f26-8815-08e38e4e0204 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:32.315593 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-e45b6129-b421-4f26-8815-08e38e4e0204 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:32.323751 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1395/2792] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:38:32.330918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b55a3f-8b87-490f-a4dd-f5b25c3bb1f6 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:32.343748 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b55a3f-8b87-490f-a4dd-f5b25c3bb1f6 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:32.349018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50326c14-75e4-41d1-977b-64dda6828724 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:32.349605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50326c14-75e4-41d1-977b-64dda6828724 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:32.349942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50326c14-75e4-41d1-977b-64dda6828724 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:32.350219 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50326c14-75e4-41d1-977b-64dda6828724 admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:32.350871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50326c14-75e4-41d1-977b-64dda6828724 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:38:32.351175 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50326c14-75e4-41d1-977b-64dda6828724 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:32.352812 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50326c14-75e4-41d1-977b-64dda6828724 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:32.365811 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1398/2793] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:38:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:38:32.370611 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45b6129-b421-4f26-8815-08e38e4e0204 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:32.380627 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45b6129-b421-4f26-8815-08e38e4e0204 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:32.385346 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21188d10-278a-44c9-a342-131893f30836 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:32.385960 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21188d10-278a-44c9-a342-131893f30836 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:32.386234 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21188d10-278a-44c9-a342-131893f30836 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:32.386504 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21188d10-278a-44c9-a342-131893f30836 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:32.387176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21188d10-278a-44c9-a342-131893f30836 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:32.389101 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21188d10-278a-44c9-a342-131893f30836 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:32.419154 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1396/2794] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:32 2022] POST /identity/v3/projects => generated 387 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:32.422803 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50326c14-75e4-41d1-977b-64dda6828724 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:32.433744 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50326c14-75e4-41d1-977b-64dda6828724 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:32.478759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:32.498492 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdcd418-b667-47d3-b09a-7a5a08b20044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:32.499136 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdcd418-b667-47d3-b09a-7a5a08b20044 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:32.499420 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdcd418-b667-47d3-b09a-7a5a08b20044 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:32.499695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdcd418-b667-47d3-b09a-7a5a08b20044 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:32.500380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdcd418-b667-47d3-b09a-7a5a08b20044 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:32.502296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdcd418-b667-47d3-b09a-7a5a08b20044 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:32.527305 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1399/2795] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:32 2022] POST /identity/v3/users => generated 294 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:32.531269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21188d10-278a-44c9-a342-131893f30836 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:32.542682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21188d10-278a-44c9-a342-131893f30836 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:32.547880 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf15ea06-6b4f-4b6c-a34e-f1e6025c4b8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:32.548526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf15ea06-6b4f-4b6c-a34e-f1e6025c4b8c admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:32.548935 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf15ea06-6b4f-4b6c-a34e-f1e6025c4b8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:32.549215 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf15ea06-6b4f-4b6c-a34e-f1e6025c4b8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:32.549888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf15ea06-6b4f-4b6c-a34e-f1e6025c4b8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:32.551473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf15ea06-6b4f-4b6c-a34e-f1e6025c4b8c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:32.562501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1397/2796] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:32 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:32.566976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdcd418-b667-47d3-b09a-7a5a08b20044 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:32.577553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdcd418-b667-47d3-b09a-7a5a08b20044 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:32.582903 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf1c257-b61d-4067-83ad-64bd3509b2ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:32.583734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf1c257-b61d-4067-83ad-64bd3509b2ff admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:32.584013 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf1c257-b61d-4067-83ad-64bd3509b2ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:32.584285 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf1c257-b61d-4067-83ad-64bd3509b2ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:32.584962 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf1c257-b61d-4067-83ad-64bd3509b2ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:32.586555 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf1c257-b61d-4067-83ad-64bd3509b2ff admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:32.601551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1400/2797] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:32 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:32.602961 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf15ea06-6b4f-4b6c-a34e-f1e6025c4b8c admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:32.614194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf15ea06-6b4f-4b6c-a34e-f1e6025c4b8c admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:32.619083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37736641-6047-445d-b319-92598161c30b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:32.619734 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37736641-6047-445d-b319-92598161c30b admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:32.620014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37736641-6047-445d-b319-92598161c30b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:32.620289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37736641-6047-445d-b319-92598161c30b admin admin] PATH_INFO: `/v3/projects/406e9eec2a054e46aad5c0b63e5b3123/users/f9d4ca38fa15462e847f7d1cd7277796/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:32.637782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37736641-6047-445d-b319-92598161c30b admin admin] RBAC: Authorizing `identity:create_grant(project_id=406e9eec2a054e46aad5c0b63e5b3123, user_id=f9d4ca38fa15462e847f7d1cd7277796, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:32.640873 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37736641-6047-445d-b319-92598161c30b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:32.657140 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1398/2798] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:32 2022] PUT /identity/v3/projects/406e9eec2a054e46aad5c0b63e5b3123/users/f9d4ca38fa15462e847f7d1cd7277796/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:32.669892 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c230becb-47f8-4314-a229-4849f32e3ad6 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:32.670251 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c230becb-47f8-4314-a229-4849f32e3ad6 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:32.670534 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c230becb-47f8-4314-a229-4849f32e3ad6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:32.700636 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-c230becb-47f8-4314-a229-4849f32e3ad6 None None] MFA Rules not processed for user `f9d4ca38fa15462e847f7d1cd7277796`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:32.701885 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c230becb-47f8-4314-a229-4849f32e3ad6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:32.784530 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1401/2799] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:32 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:32.789888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37736641-6047-445d-b319-92598161c30b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:32.825824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:32.864304 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37736641-6047-445d-b319-92598161c30b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:32.869932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9d67ac-d67d-49ba-be86-58ccba09ab43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:32.870543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9d67ac-d67d-49ba-be86-58ccba09ab43 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:32.870817 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9d67ac-d67d-49ba-be86-58ccba09ab43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:32.871088 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9d67ac-d67d-49ba-be86-58ccba09ab43 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:32.871733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9d67ac-d67d-49ba-be86-58ccba09ab43 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:32.873673 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9d67ac-d67d-49ba-be86-58ccba09ab43 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:32.901806 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1399/2800] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:32 2022] POST /identity/v3/projects => generated 383 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:32.906150 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c230becb-47f8-4314-a229-4849f32e3ad6 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:32.918524 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c230becb-47f8-4314-a229-4849f32e3ad6 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:32.975888 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e005678-b23f-46f8-9bda-118916d0ea5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:32.976440 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e005678-b23f-46f8-9bda-118916d0ea5f admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:32.976814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e005678-b23f-46f8-9bda-118916d0ea5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:32.977094 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e005678-b23f-46f8-9bda-118916d0ea5f admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:32.977732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e005678-b23f-46f8-9bda-118916d0ea5f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:32.979641 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e005678-b23f-46f8-9bda-118916d0ea5f admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:33.003514 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1402/2801] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:32 2022] POST /identity/v3/users => generated 292 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:33.004702 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9d67ac-d67d-49ba-be86-58ccba09ab43 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:33.015172 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9d67ac-d67d-49ba-be86-58ccba09ab43 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:33.020014 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1df90ab-efd9-442e-a25a-1b36c63a8948 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:33.020664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1df90ab-efd9-442e-a25a-1b36c63a8948 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:33.020943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1df90ab-efd9-442e-a25a-1b36c63a8948 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:33.021269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1df90ab-efd9-442e-a25a-1b36c63a8948 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:33.021905 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1df90ab-efd9-442e-a25a-1b36c63a8948 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:33.023500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1df90ab-efd9-442e-a25a-1b36c63a8948 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:33.034517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1400/2802] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:33 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:38:33.039522 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e005678-b23f-46f8-9bda-118916d0ea5f admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:33.049721 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e005678-b23f-46f8-9bda-118916d0ea5f admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:33.055222 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73376e8e-14f9-420f-91f6-3e1c647db1e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:33.055813 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73376e8e-14f9-420f-91f6-3e1c647db1e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:33.056093 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73376e8e-14f9-420f-91f6-3e1c647db1e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:33.056397 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73376e8e-14f9-420f-91f6-3e1c647db1e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:33.057063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73376e8e-14f9-420f-91f6-3e1c647db1e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:33.058649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73376e8e-14f9-420f-91f6-3e1c647db1e5 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:33.068752 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1403/2803] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:33 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:33.073787 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1df90ab-efd9-442e-a25a-1b36c63a8948 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:33.084016 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1df90ab-efd9-442e-a25a-1b36c63a8948 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:33.088745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe9a32e-d812-4dfc-b668-abf6023e4319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:33.089375 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe9a32e-d812-4dfc-b668-abf6023e4319 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:33.089655 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe9a32e-d812-4dfc-b668-abf6023e4319 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:33.089925 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe9a32e-d812-4dfc-b668-abf6023e4319 admin admin] PATH_INFO: `/v3/projects/b7a3de6ae9f247b6a12bae617efccd17/users/3c89448c49fa41a7832d9abc9b335162/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:33.105013 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe9a32e-d812-4dfc-b668-abf6023e4319 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b7a3de6ae9f247b6a12bae617efccd17, user_id=3c89448c49fa41a7832d9abc9b335162, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:33.107634 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe9a32e-d812-4dfc-b668-abf6023e4319 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:33.121012 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1401/2804] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:33 2022] PUT /identity/v3/projects/b7a3de6ae9f247b6a12bae617efccd17/users/3c89448c49fa41a7832d9abc9b335162/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:33.131443 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4fbd8b-ea7e-4615-bfb3-e5d8671b3c3a None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:33.131927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4fbd8b-ea7e-4615-bfb3-e5d8671b3c3a None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:33.132342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4fbd8b-ea7e-4615-bfb3-e5d8671b3c3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:33.161822 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-df4fbd8b-ea7e-4615-bfb3-e5d8671b3c3a None None] MFA Rules not processed for user `3c89448c49fa41a7832d9abc9b335162`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:33.163069 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-df4fbd8b-ea7e-4615-bfb3-e5d8671b3c3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:33.182046 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:33.232335 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1404/2805] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:33 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:33.236943 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe9a32e-d812-4dfc-b668-abf6023e4319 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:33.301047 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe9a32e-d812-4dfc-b668-abf6023e4319 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:33.305831 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a342809-7af4-4096-9ed6-263a656175f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:33.306481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a342809-7af4-4096-9ed6-263a656175f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:33.306763 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a342809-7af4-4096-9ed6-263a656175f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:33.307036 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a342809-7af4-4096-9ed6-263a656175f7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:33.307679 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a342809-7af4-4096-9ed6-263a656175f7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:33.309898 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a342809-7af4-4096-9ed6-263a656175f7 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:33.338681 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1402/2806] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:33 2022] POST /identity/v3/projects => generated 385 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:38:33.343448 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4fbd8b-ea7e-4615-bfb3-e5d8671b3c3a None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:33.354582 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4fbd8b-ea7e-4615-bfb3-e5d8671b3c3a None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:33.411925 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b768705c-2c17-4d1a-b189-26cb9bad2e30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:33.412558 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b768705c-2c17-4d1a-b189-26cb9bad2e30 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:33.412842 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b768705c-2c17-4d1a-b189-26cb9bad2e30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:33.413117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b768705c-2c17-4d1a-b189-26cb9bad2e30 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:33.413758 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b768705c-2c17-4d1a-b189-26cb9bad2e30 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:33.415678 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b768705c-2c17-4d1a-b189-26cb9bad2e30 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:33.438861 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1405/2807] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:33 2022] POST /identity/v3/users => generated 299 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:33.446497 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a342809-7af4-4096-9ed6-263a656175f7 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:33.459994 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a342809-7af4-4096-9ed6-263a656175f7 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:33.464840 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35abc37f-743a-4a5d-90d9-b19a55b80d7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:33.465442 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35abc37f-743a-4a5d-90d9-b19a55b80d7b admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:33.465720 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35abc37f-743a-4a5d-90d9-b19a55b80d7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:33.465991 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35abc37f-743a-4a5d-90d9-b19a55b80d7b admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:33.466631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35abc37f-743a-4a5d-90d9-b19a55b80d7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:33.468217 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35abc37f-743a-4a5d-90d9-b19a55b80d7b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:33.480451 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1403/2808] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:33 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:33.484884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b768705c-2c17-4d1a-b189-26cb9bad2e30 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:33.495445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b768705c-2c17-4d1a-b189-26cb9bad2e30 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:33.500139 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d021dc36-39e1-46c7-9ccb-259774f64817 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:33.500852 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d021dc36-39e1-46c7-9ccb-259774f64817 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:33.501135 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d021dc36-39e1-46c7-9ccb-259774f64817 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:33.501414 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d021dc36-39e1-46c7-9ccb-259774f64817 admin admin] PATH_INFO: `/v3/projects/e2cbd1ca833e4c2280823c7bae4751e9/users/a4103eda469849509d5618ce876b2f4b/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:33.518523 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d021dc36-39e1-46c7-9ccb-259774f64817 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2cbd1ca833e4c2280823c7bae4751e9, user_id=a4103eda469849509d5618ce876b2f4b, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:33.522246 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d021dc36-39e1-46c7-9ccb-259774f64817 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:33.537054 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1406/2809] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:33 2022] PUT /identity/v3/projects/e2cbd1ca833e4c2280823c7bae4751e9/users/a4103eda469849509d5618ce876b2f4b/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:33.542906 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35abc37f-743a-4a5d-90d9-b19a55b80d7b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:33.556564 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35abc37f-743a-4a5d-90d9-b19a55b80d7b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:33.615270 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:33.625561 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4991b064-f4cd-4786-b089-df1a449f976a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:33.626178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4991b064-f4cd-4786-b089-df1a449f976a admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:33.626456 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4991b064-f4cd-4786-b089-df1a449f976a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:33.626726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4991b064-f4cd-4786-b089-df1a449f976a admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:33.627371 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4991b064-f4cd-4786-b089-df1a449f976a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:33.629612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4991b064-f4cd-4786-b089-df1a449f976a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:33.643932 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1404/2810] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:33 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:33.650572 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d021dc36-39e1-46c7-9ccb-259774f64817 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:33.663050 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d021dc36-39e1-46c7-9ccb-259774f64817 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:33.668004 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5075998-7437-424d-974d-c7bead4716a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:33.668731 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5075998-7437-424d-974d-c7bead4716a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:33.669014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5075998-7437-424d-974d-c7bead4716a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:33.669288 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5075998-7437-424d-974d-c7bead4716a1 admin admin] PATH_INFO: `/v3/projects/e2cbd1ca833e4c2280823c7bae4751e9/users/a4103eda469849509d5618ce876b2f4b/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:33.671260 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5075998-7437-424d-974d-c7bead4716a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2cbd1ca833e4c2280823c7bae4751e9, user_id=a4103eda469849509d5618ce876b2f4b, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:33.674172 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5075998-7437-424d-974d-c7bead4716a1 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:33.688198 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1407/2811] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:33 2022] PUT /identity/v3/projects/e2cbd1ca833e4c2280823c7bae4751e9/users/a4103eda469849509d5618ce876b2f4b/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:33.705573 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eb8161-8ada-4922-a3c8-5cfb3cf341b5 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:33.705878 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eb8161-8ada-4922-a3c8-5cfb3cf341b5 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:33.706301 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eb8161-8ada-4922-a3c8-5cfb3cf341b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:33.741914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-e2eb8161-8ada-4922-a3c8-5cfb3cf341b5 None None] MFA Rules not processed for user `a4103eda469849509d5618ce876b2f4b`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:33.742972 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-e2eb8161-8ada-4922-a3c8-5cfb3cf341b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:33.828720 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1405/2812] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:33 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:33.841420 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5075998-7437-424d-974d-c7bead4716a1 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:33.845990 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-f5075998-7437-424d-974d-c7bead4716a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:33.927537 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5075998-7437-424d-974d-c7bead4716a1 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:33.931273 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f598a3c8-fc95-4b24-9691-b11076db3806 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:33.931886 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f598a3c8-fc95-4b24-9691-b11076db3806 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:33.932166 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f598a3c8-fc95-4b24-9691-b11076db3806 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:33.932484 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f598a3c8-fc95-4b24-9691-b11076db3806 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:33.946168 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f598a3c8-fc95-4b24-9691-b11076db3806 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:33.947787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f598a3c8-fc95-4b24-9691-b11076db3806 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:33.959043 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1408/2813] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:33 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:34.564249 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2eb8161-8ada-4922-a3c8-5cfb3cf341b5 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:34.566888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-e2eb8161-8ada-4922-a3c8-5cfb3cf341b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:34.579218 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2eb8161-8ada-4922-a3c8-5cfb3cf341b5 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:34.583477 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce3187c-62ee-4b9f-84c6-2af3bcf39ca5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:34.584056 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce3187c-62ee-4b9f-84c6-2af3bcf39ca5 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:34.584366 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce3187c-62ee-4b9f-84c6-2af3bcf39ca5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:34.584643 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce3187c-62ee-4b9f-84c6-2af3bcf39ca5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:34.646287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce3187c-62ee-4b9f-84c6-2af3bcf39ca5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:34.647784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce3187c-62ee-4b9f-84c6-2af3bcf39ca5 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:34.658046 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1406/2814] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:34 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:41.990845 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f598a3c8-fc95-4b24-9691-b11076db3806 service cinder] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:42.004849 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f598a3c8-fc95-4b24-9691-b11076db3806 service cinder] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:42.009909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080f3146-9d44-4dde-88b4-248a58fc0770 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:42.010735 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080f3146-9d44-4dde-88b4-248a58fc0770 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:42.011085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080f3146-9d44-4dde-88b4-248a58fc0770 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:42.011402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080f3146-9d44-4dde-88b4-248a58fc0770 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:42.025944 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:42.082882 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080f3146-9d44-4dde-88b4-248a58fc0770 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:42.084626 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080f3146-9d44-4dde-88b4-248a58fc0770 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:42.094110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1409/2815] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:41 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:38:44.354462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce3187c-62ee-4b9f-84c6-2af3bcf39ca5 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:44.422269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce3187c-62ee-4b9f-84c6-2af3bcf39ca5 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:44.427419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6012e86a-076d-4dfb-90c7-e68da7f31b3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:44.428050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6012e86a-076d-4dfb-90c7-e68da7f31b3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:44.428349 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6012e86a-076d-4dfb-90c7-e68da7f31b3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:44.428627 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6012e86a-076d-4dfb-90c7-e68da7f31b3a admin admin] PATH_INFO: `/v3/users/f9d4ca38fa15462e847f7d1cd7277796` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:44.429762 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6012e86a-076d-4dfb-90c7-e68da7f31b3a admin admin] RBAC: Authorizing `identity:delete_user(user_id=f9d4ca38fa15462e847f7d1cd7277796)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:44.431680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6012e86a-076d-4dfb-90c7-e68da7f31b3a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:44.446097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:44.481296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6012e86a-076d-4dfb-90c7-e68da7f31b3a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f9d4ca38fa15462e847f7d1cd7277796'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:44.490826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6012e86a-076d-4dfb-90c7-e68da7f31b3a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f9d4ca38fa15462e847f7d1cd7277796'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:44.499605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6012e86a-076d-4dfb-90c7-e68da7f31b3a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f9d4ca38fa15462e847f7d1cd7277796'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:44.515277 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6012e86a-076d-4dfb-90c7-e68da7f31b3a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f9d4ca38fa15462e847f7d1cd7277796'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:44.516023 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-6012e86a-076d-4dfb-90c7-e68da7f31b3a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f9d4ca38fa15462e847f7d1cd7277796'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:44.522913 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1407/2816] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:44 2022] DELETE /identity/v3/users/f9d4ca38fa15462e847f7d1cd7277796 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:44.528211 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7582157f-dac5-4527-b70c-a4ad1ddcb610 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:44.528780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7582157f-dac5-4527-b70c-a4ad1ddcb610 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:44.528884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7582157f-dac5-4527-b70c-a4ad1ddcb610 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:44.550318 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-7582157f-dac5-4527-b70c-a4ad1ddcb610 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:44.551767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7582157f-dac5-4527-b70c-a4ad1ddcb610 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:44.616552 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1410/2817] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:44.626777 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6012e86a-076d-4dfb-90c7-e68da7f31b3a admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:44.628405 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-6012e86a-076d-4dfb-90c7-e68da7f31b3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:44.696635 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6012e86a-076d-4dfb-90c7-e68da7f31b3a admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:44.700344 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9efbbe-62ab-42a5-9acc-1c1e148a783e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:44.700962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9efbbe-62ab-42a5-9acc-1c1e148a783e service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:44.701238 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9efbbe-62ab-42a5-9acc-1c1e148a783e service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:44.701506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9efbbe-62ab-42a5-9acc-1c1e148a783e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:44.712522 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9efbbe-62ab-42a5-9acc-1c1e148a783e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:44.714021 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9efbbe-62ab-42a5-9acc-1c1e148a783e service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:44.725921 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1408/2818] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:45.008064 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7582157f-dac5-4527-b70c-a4ad1ddcb610 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:45.009636 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7582157f-dac5-4527-b70c-a4ad1ddcb610 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:45.023708 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7582157f-dac5-4527-b70c-a4ad1ddcb610 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:45.029264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c90883-2099-48d3-8bfa-6f4d7ca65823 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:45.029734 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c90883-2099-48d3-8bfa-6f4d7ca65823 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:45.030076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c90883-2099-48d3-8bfa-6f4d7ca65823 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:45.030386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c90883-2099-48d3-8bfa-6f4d7ca65823 admin admin] PATH_INFO: `/v3/projects/406e9eec2a054e46aad5c0b63e5b3123` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:45.031476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c90883-2099-48d3-8bfa-6f4d7ca65823 admin admin] RBAC: Authorizing `identity:delete_project(project_id=406e9eec2a054e46aad5c0b63e5b3123)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:45.033432 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c90883-2099-48d3-8bfa-6f4d7ca65823 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:45.086442 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-50c90883-2099-48d3-8bfa-6f4d7ca65823 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '406e9eec2a054e46aad5c0b63e5b3123', 'request_id': 'req-50c90883-2099-48d3-8bfa-6f4d7ca65823'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.093642 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-50c90883-2099-48d3-8bfa-6f4d7ca65823 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '406e9eec2a054e46aad5c0b63e5b3123', 'request_id': 'req-50c90883-2099-48d3-8bfa-6f4d7ca65823'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.100606 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-50c90883-2099-48d3-8bfa-6f4d7ca65823 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '406e9eec2a054e46aad5c0b63e5b3123', 'request_id': 'req-50c90883-2099-48d3-8bfa-6f4d7ca65823'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.114927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-50c90883-2099-48d3-8bfa-6f4d7ca65823 admin admin] The token cache is being invalidate because project 406e9eec2a054e46aad5c0b63e5b3123 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:45.115216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-50c90883-2099-48d3-8bfa-6f4d7ca65823 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.115884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-50c90883-2099-48d3-8bfa-6f4d7ca65823 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.117407 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1411/2819] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:45 2022] DELETE /identity/v3/projects/406e9eec2a054e46aad5c0b63e5b3123 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:45.121429 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9efbbe-62ab-42a5-9acc-1c1e148a783e service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:45.131442 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9efbbe-62ab-42a5-9acc-1c1e148a783e service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:45.191621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ca3ddc-6ebb-46c6-89f1-f1658ab4c172 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:45.192249 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ca3ddc-6ebb-46c6-89f1-f1658ab4c172 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:45.192569 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ca3ddc-6ebb-46c6-89f1-f1658ab4c172 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:45.192839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ca3ddc-6ebb-46c6-89f1-f1658ab4c172 admin admin] PATH_INFO: `/v3/users/3c89448c49fa41a7832d9abc9b335162` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:45.194062 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ca3ddc-6ebb-46c6-89f1-f1658ab4c172 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3c89448c49fa41a7832d9abc9b335162)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:45.195989 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ca3ddc-6ebb-46c6-89f1-f1658ab4c172 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:45.246184 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:45.259970 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-32ca3ddc-6ebb-46c6-89f1-f1658ab4c172 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3c89448c49fa41a7832d9abc9b335162'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.271471 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-32ca3ddc-6ebb-46c6-89f1-f1658ab4c172 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3c89448c49fa41a7832d9abc9b335162'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.279291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-32ca3ddc-6ebb-46c6-89f1-f1658ab4c172 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3c89448c49fa41a7832d9abc9b335162'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.293041 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-32ca3ddc-6ebb-46c6-89f1-f1658ab4c172 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3c89448c49fa41a7832d9abc9b335162'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.293903 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-32ca3ddc-6ebb-46c6-89f1-f1658ab4c172 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3c89448c49fa41a7832d9abc9b335162'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.300745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1409/2820] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:45 2022] DELETE /identity/v3/users/3c89448c49fa41a7832d9abc9b335162 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:45.596569 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c90883-2099-48d3-8bfa-6f4d7ca65823 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:45.600331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-50c90883-2099-48d3-8bfa-6f4d7ca65823 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:45.616680 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c90883-2099-48d3-8bfa-6f4d7ca65823 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:45.653847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:45.680638 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:45.681287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:45.681571 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:45.681850 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3 admin admin] PATH_INFO: `/v3/projects/b7a3de6ae9f247b6a12bae617efccd17` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:45.683016 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b7a3de6ae9f247b6a12bae617efccd17)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:45.685003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:45.737205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b7a3de6ae9f247b6a12bae617efccd17', 'request_id': 'req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.745207 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b7a3de6ae9f247b6a12bae617efccd17', 'request_id': 'req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.751199 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b7a3de6ae9f247b6a12bae617efccd17', 'request_id': 'req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.764513 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3 admin admin] The token cache is being invalidate because project b7a3de6ae9f247b6a12bae617efccd17 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:45.764806 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.765457 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.766963 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1412/2821] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:45 2022] DELETE /identity/v3/projects/b7a3de6ae9f247b6a12bae617efccd17 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:45.778003 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ca3ddc-6ebb-46c6-89f1-f1658ab4c172 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:45.779513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-32ca3ddc-6ebb-46c6-89f1-f1658ab4c172 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:45.801753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ca3ddc-6ebb-46c6-89f1-f1658ab4c172 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:45.866549 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5503615-057b-41d0-9d4c-a545747d736b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:45.867264 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5503615-057b-41d0-9d4c-a545747d736b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:45.867582 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5503615-057b-41d0-9d4c-a545747d736b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:45.867895 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5503615-057b-41d0-9d4c-a545747d736b admin admin] PATH_INFO: `/v3/users/a4103eda469849509d5618ce876b2f4b` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:45.869157 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5503615-057b-41d0-9d4c-a545747d736b admin admin] RBAC: Authorizing `identity:delete_user(user_id=a4103eda469849509d5618ce876b2f4b)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:45.871585 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5503615-057b-41d0-9d4c-a545747d736b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:45.922443 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-f5503615-057b-41d0-9d4c-a545747d736b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a4103eda469849509d5618ce876b2f4b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.930815 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-f5503615-057b-41d0-9d4c-a545747d736b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a4103eda469849509d5618ce876b2f4b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.939977 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-f5503615-057b-41d0-9d4c-a545747d736b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a4103eda469849509d5618ce876b2f4b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.953324 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-f5503615-057b-41d0-9d4c-a545747d736b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a4103eda469849509d5618ce876b2f4b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.954035 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-f5503615-057b-41d0-9d4c-a545747d736b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a4103eda469849509d5618ce876b2f4b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:45.960543 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1410/2822] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:45 2022] DELETE /identity/v3/users/a4103eda469849509d5618ce876b2f4b => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:46.238499 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:46.240159 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:46.253814 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f49856e-931c-4b91-a6f0-cad6b4aa87b3 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:46.311178 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745a6599-f461-4995-8517-118b8240dd6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:46.311816 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a6599-f461-4995-8517-118b8240dd6d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:46.312101 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a6599-f461-4995-8517-118b8240dd6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:46.312405 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a6599-f461-4995-8517-118b8240dd6d admin admin] PATH_INFO: `/v3/projects/e2cbd1ca833e4c2280823c7bae4751e9` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:46.313579 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745a6599-f461-4995-8517-118b8240dd6d admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2cbd1ca833e4c2280823c7bae4751e9)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:46.316066 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745a6599-f461-4995-8517-118b8240dd6d admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:46.364022 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-745a6599-f461-4995-8517-118b8240dd6d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2cbd1ca833e4c2280823c7bae4751e9', 'request_id': 'req-745a6599-f461-4995-8517-118b8240dd6d'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:46.370485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-745a6599-f461-4995-8517-118b8240dd6d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2cbd1ca833e4c2280823c7bae4751e9', 'request_id': 'req-745a6599-f461-4995-8517-118b8240dd6d'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:46.376581 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-745a6599-f461-4995-8517-118b8240dd6d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2cbd1ca833e4c2280823c7bae4751e9', 'request_id': 'req-745a6599-f461-4995-8517-118b8240dd6d'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:46.384701 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:46.392887 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-745a6599-f461-4995-8517-118b8240dd6d admin admin] The token cache is being invalidate because project e2cbd1ca833e4c2280823c7bae4751e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:46.393349 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-745a6599-f461-4995-8517-118b8240dd6d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:46.394218 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-745a6599-f461-4995-8517-118b8240dd6d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:46.395693 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1413/2823] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:46 2022] DELETE /identity/v3/projects/e2cbd1ca833e4c2280823c7bae4751e9 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:46.401877 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5150f5f-33bf-4197-9708-185c68c6e567 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:46.402237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5150f5f-33bf-4197-9708-185c68c6e567 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:46.402441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5150f5f-33bf-4197-9708-185c68c6e567 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:46.422063 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-a5150f5f-33bf-4197-9708-185c68c6e567 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:46.423270 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a5150f5f-33bf-4197-9708-185c68c6e567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:46.455114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:46.487060 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1411/2824] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:46.496257 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33488059-fdcd-4ec5-bc45-2ad9eaa04bd9 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:46.496587 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33488059-fdcd-4ec5-bc45-2ad9eaa04bd9 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:46.496891 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33488059-fdcd-4ec5-bc45-2ad9eaa04bd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:46.516871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-33488059-fdcd-4ec5-bc45-2ad9eaa04bd9 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:46.518080 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-33488059-fdcd-4ec5-bc45-2ad9eaa04bd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:46.528853 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1414/2825] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:38:46.533291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5150f5f-33bf-4197-9708-185c68c6e567 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:46.543568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5150f5f-33bf-4197-9708-185c68c6e567 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:46.548332 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2c38cc-5b66-4f8b-9cb9-57e9f2f5e222 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:46.548924 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2c38cc-5b66-4f8b-9cb9-57e9f2f5e222 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:46.549203 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2c38cc-5b66-4f8b-9cb9-57e9f2f5e222 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:46.549478 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2c38cc-5b66-4f8b-9cb9-57e9f2f5e222 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:46.550793 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2c38cc-5b66-4f8b-9cb9-57e9f2f5e222 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:38:46.551110 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2c38cc-5b66-4f8b-9cb9-57e9f2f5e222 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:46.552829 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2c38cc-5b66-4f8b-9cb9-57e9f2f5e222 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:46.565412 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1412/2826] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:38:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:38:46.569798 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33488059-fdcd-4ec5-bc45-2ad9eaa04bd9 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:46.579066 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33488059-fdcd-4ec5-bc45-2ad9eaa04bd9 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:46.583963 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ddf50a-2606-4102-b725-5756ae28351c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:46.584602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ddf50a-2606-4102-b725-5756ae28351c admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:46.584884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ddf50a-2606-4102-b725-5756ae28351c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:46.585155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ddf50a-2606-4102-b725-5756ae28351c admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:46.585785 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ddf50a-2606-4102-b725-5756ae28351c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:46.587691 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ddf50a-2606-4102-b725-5756ae28351c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:46.616670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1415/2827] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:46 2022] POST /identity/v3/projects => generated 359 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:46.621011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2c38cc-5b66-4f8b-9cb9-57e9f2f5e222 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:46.631701 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2c38cc-5b66-4f8b-9cb9-57e9f2f5e222 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:46.687858 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b2fb18-e386-4ef6-95ae-0249139abc73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:46.688487 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b2fb18-e386-4ef6-95ae-0249139abc73 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:46.688766 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b2fb18-e386-4ef6-95ae-0249139abc73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:46.689042 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b2fb18-e386-4ef6-95ae-0249139abc73 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:46.689664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b2fb18-e386-4ef6-95ae-0249139abc73 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:46.692252 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b2fb18-e386-4ef6-95ae-0249139abc73 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:46.712813 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1413/2828] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:46 2022] POST /identity/v3/users => generated 280 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:46.716969 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ddf50a-2606-4102-b725-5756ae28351c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:46.727656 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ddf50a-2606-4102-b725-5756ae28351c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:46.732153 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967b3122-e701-4885-b544-3b3bcd61f359 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:46.732769 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967b3122-e701-4885-b544-3b3bcd61f359 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:46.733044 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967b3122-e701-4885-b544-3b3bcd61f359 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:46.733314 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967b3122-e701-4885-b544-3b3bcd61f359 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:46.733947 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967b3122-e701-4885-b544-3b3bcd61f359 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:46.735517 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967b3122-e701-4885-b544-3b3bcd61f359 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:46.746041 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1416/2829] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:46.749992 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b2fb18-e386-4ef6-95ae-0249139abc73 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:46.760277 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b2fb18-e386-4ef6-95ae-0249139abc73 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:46.764879 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814734a8-a46f-4b13-8111-c049965b859a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:46.765475 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814734a8-a46f-4b13-8111-c049965b859a admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:46.765753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814734a8-a46f-4b13-8111-c049965b859a admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:46.766026 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814734a8-a46f-4b13-8111-c049965b859a admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:46.766667 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814734a8-a46f-4b13-8111-c049965b859a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:46.768243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814734a8-a46f-4b13-8111-c049965b859a admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:46.780176 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1414/2830] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:46 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:46.785079 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967b3122-e701-4885-b544-3b3bcd61f359 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:46.795036 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967b3122-e701-4885-b544-3b3bcd61f359 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:46.799749 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c60da0-c69b-425f-b31b-cd2a5b958046 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:46.800633 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c60da0-c69b-425f-b31b-cd2a5b958046 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:46.800914 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c60da0-c69b-425f-b31b-cd2a5b958046 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:46.801194 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c60da0-c69b-425f-b31b-cd2a5b958046 admin admin] PATH_INFO: `/v3/projects/5aa08123af5c4418899cfd72ecd6d96a/users/21b9ee1c809c40058067f7bcadfab6f6/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:46.816546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c60da0-c69b-425f-b31b-cd2a5b958046 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5aa08123af5c4418899cfd72ecd6d96a, user_id=21b9ee1c809c40058067f7bcadfab6f6, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:46.819185 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c60da0-c69b-425f-b31b-cd2a5b958046 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:46.833263 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1417/2831] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:46 2022] PUT /identity/v3/projects/5aa08123af5c4418899cfd72ecd6d96a/users/21b9ee1c809c40058067f7bcadfab6f6/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:46.844717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a39bcc-20e9-467d-bfce-2de36efa4926 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:46.845001 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a39bcc-20e9-467d-bfce-2de36efa4926 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:46.845279 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a39bcc-20e9-467d-bfce-2de36efa4926 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:46.876397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-21a39bcc-20e9-467d-bfce-2de36efa4926 None None] MFA Rules not processed for user `21b9ee1c809c40058067f7bcadfab6f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:46.877631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-21a39bcc-20e9-467d-bfce-2de36efa4926 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:46.951328 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1415/2832] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:46 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:38:46.989993 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c60da0-c69b-425f-b31b-cd2a5b958046 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:46.992015 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-78c60da0-c69b-425f-b31b-cd2a5b958046 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:47.055538 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c60da0-c69b-425f-b31b-cd2a5b958046 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:47.059297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d5af02-387f-4a1a-a672-22dcf2482659 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:47.059943 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d5af02-387f-4a1a-a672-22dcf2482659 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:47.060224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d5af02-387f-4a1a-a672-22dcf2482659 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:47.060551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d5af02-387f-4a1a-a672-22dcf2482659 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:47.071052 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d5af02-387f-4a1a-a672-22dcf2482659 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:47.072513 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d5af02-387f-4a1a-a672-22dcf2482659 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:47.081999 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1418/2833] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:46 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:47.098102 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a39bcc-20e9-467d-bfce-2de36efa4926 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:47.131208 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:47.179790 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a39bcc-20e9-467d-bfce-2de36efa4926 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:47.184551 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e25a699-9668-4b6a-848d-1b141cec9ec6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:47.185187 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e25a699-9668-4b6a-848d-1b141cec9ec6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:47.185463 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e25a699-9668-4b6a-848d-1b141cec9ec6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:47.185733 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e25a699-9668-4b6a-848d-1b141cec9ec6 admin admin] PATH_INFO: `/v3/users/21b9ee1c809c40058067f7bcadfab6f6` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:47.186881 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e25a699-9668-4b6a-848d-1b141cec9ec6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=21b9ee1c809c40058067f7bcadfab6f6)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:47.188859 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e25a699-9668-4b6a-848d-1b141cec9ec6 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:47.243993 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-0e25a699-9668-4b6a-848d-1b141cec9ec6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21b9ee1c809c40058067f7bcadfab6f6'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:47.251946 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-0e25a699-9668-4b6a-848d-1b141cec9ec6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21b9ee1c809c40058067f7bcadfab6f6'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:47.260157 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-0e25a699-9668-4b6a-848d-1b141cec9ec6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21b9ee1c809c40058067f7bcadfab6f6'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:47.275022 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-0e25a699-9668-4b6a-848d-1b141cec9ec6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21b9ee1c809c40058067f7bcadfab6f6'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:47.277020 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-0e25a699-9668-4b6a-848d-1b141cec9ec6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21b9ee1c809c40058067f7bcadfab6f6'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:47.283462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1416/2834] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:47 2022] DELETE /identity/v3/users/21b9ee1c809c40058067f7bcadfab6f6 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:38:47.288804 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9f37fd-8187-4b45-bab9-bfdc2a6d2d1b None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:47.289093 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9f37fd-8187-4b45-bab9-bfdc2a6d2d1b None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:47.289398 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9f37fd-8187-4b45-bab9-bfdc2a6d2d1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:47.310966 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-fd9f37fd-8187-4b45-bab9-bfdc2a6d2d1b None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:47.312212 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-fd9f37fd-8187-4b45-bab9-bfdc2a6d2d1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:47.379368 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1419/2835] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:47.389588 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e25a699-9668-4b6a-848d-1b141cec9ec6 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:47.391083 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-0e25a699-9668-4b6a-848d-1b141cec9ec6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:47.459210 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e25a699-9668-4b6a-848d-1b141cec9ec6 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:47.463122 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85eb525-8cd1-42fd-9180-472b3fe889a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:47.463845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85eb525-8cd1-42fd-9180-472b3fe889a6 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:47.464011 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85eb525-8cd1-42fd-9180-472b3fe889a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:47.464335 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85eb525-8cd1-42fd-9180-472b3fe889a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:47.475648 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85eb525-8cd1-42fd-9180-472b3fe889a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:47.477173 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85eb525-8cd1-42fd-9180-472b3fe889a6 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:47.488676 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1417/2836] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:47.772161 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9f37fd-8187-4b45-bab9-bfdc2a6d2d1b None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:47.773711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-fd9f37fd-8187-4b45-bab9-bfdc2a6d2d1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:47.788457 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9f37fd-8187-4b45-bab9-bfdc2a6d2d1b None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:47.794893 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9980c40-523a-4f1b-8b16-ea7f3a79f515 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:47.795803 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9980c40-523a-4f1b-8b16-ea7f3a79f515 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:47.796085 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9980c40-523a-4f1b-8b16-ea7f3a79f515 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:47.796396 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9980c40-523a-4f1b-8b16-ea7f3a79f515 admin admin] PATH_INFO: `/v3/projects/5aa08123af5c4418899cfd72ecd6d96a` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:47.797956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9980c40-523a-4f1b-8b16-ea7f3a79f515 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5aa08123af5c4418899cfd72ecd6d96a)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:47.800715 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9980c40-523a-4f1b-8b16-ea7f3a79f515 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:47.837586 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:47.853784 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b9980c40-523a-4f1b-8b16-ea7f3a79f515 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5aa08123af5c4418899cfd72ecd6d96a', 'request_id': 'req-b9980c40-523a-4f1b-8b16-ea7f3a79f515'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:47.860332 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b9980c40-523a-4f1b-8b16-ea7f3a79f515 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5aa08123af5c4418899cfd72ecd6d96a', 'request_id': 'req-b9980c40-523a-4f1b-8b16-ea7f3a79f515'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:47.866305 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b9980c40-523a-4f1b-8b16-ea7f3a79f515 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5aa08123af5c4418899cfd72ecd6d96a', 'request_id': 'req-b9980c40-523a-4f1b-8b16-ea7f3a79f515'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:47.880360 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b9980c40-523a-4f1b-8b16-ea7f3a79f515 admin admin] The token cache is being invalidate because project 5aa08123af5c4418899cfd72ecd6d96a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:47.880773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b9980c40-523a-4f1b-8b16-ea7f3a79f515 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:47.881564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-b9980c40-523a-4f1b-8b16-ea7f3a79f515 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:47.883143 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1420/2837] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:47 2022] DELETE /identity/v3/projects/5aa08123af5c4418899cfd72ecd6d96a => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:38:47.888243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfee4f9-5689-4df0-a7a0-6dfdefc0c072 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:47.888791 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfee4f9-5689-4df0-a7a0-6dfdefc0c072 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:47.888916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfee4f9-5689-4df0-a7a0-6dfdefc0c072 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:47.912506 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-ccfee4f9-5689-4df0-a7a0-6dfdefc0c072 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:47.913874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ccfee4f9-5689-4df0-a7a0-6dfdefc0c072 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:47.962260 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:47.979237 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1418/2838] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:38:47.988475 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9833a970-182a-4de1-a2bc-11eaa15cfbee None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:47.988788 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9833a970-182a-4de1-a2bc-11eaa15cfbee None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:47.989063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9833a970-182a-4de1-a2bc-11eaa15cfbee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:48.009747 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-9833a970-182a-4de1-a2bc-11eaa15cfbee None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:48.010981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9833a970-182a-4de1-a2bc-11eaa15cfbee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:48.019927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1421/2839] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:48.024277 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfee4f9-5689-4df0-a7a0-6dfdefc0c072 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:48.035119 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfee4f9-5689-4df0-a7a0-6dfdefc0c072 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:48.040793 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cf2ffc-b4b1-4117-9bdb-bad1a44f5e96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:48.041407 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cf2ffc-b4b1-4117-9bdb-bad1a44f5e96 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:48.041686 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cf2ffc-b4b1-4117-9bdb-bad1a44f5e96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:48.041966 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cf2ffc-b4b1-4117-9bdb-bad1a44f5e96 admin admin] PATH_INFO: `/v3/domains` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:48.042639 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10cf2ffc-b4b1-4117-9bdb-bad1a44f5e96 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81461) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:38:48.042916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10cf2ffc-b4b1-4117-9bdb-bad1a44f5e96 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:48.044539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10cf2ffc-b4b1-4117-9bdb-bad1a44f5e96 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:48.057390 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1419/2840] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:38:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:38:48.061833 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9833a970-182a-4de1-a2bc-11eaa15cfbee None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:48.071995 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9833a970-182a-4de1-a2bc-11eaa15cfbee None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:48.077439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d76d072-d081-497d-bb59-03f8eb779afc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:48.078049 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d76d072-d081-497d-bb59-03f8eb779afc admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:48.078336 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d76d072-d081-497d-bb59-03f8eb779afc admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:48.078611 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d76d072-d081-497d-bb59-03f8eb779afc admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:48.079264 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d76d072-d081-497d-bb59-03f8eb779afc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:48.081837 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d76d072-d081-497d-bb59-03f8eb779afc admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:48.111067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1422/2841] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:48 2022] POST /identity/v3/projects => generated 383 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:38:48.115892 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cf2ffc-b4b1-4117-9bdb-bad1a44f5e96 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:48.126583 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cf2ffc-b4b1-4117-9bdb-bad1a44f5e96 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:48.191343 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e257d195-39da-40ff-a227-73a56e541f23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:48.192431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e257d195-39da-40ff-a227-73a56e541f23 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:48.192926 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e257d195-39da-40ff-a227-73a56e541f23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:48.193439 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e257d195-39da-40ff-a227-73a56e541f23 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:48.194638 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e257d195-39da-40ff-a227-73a56e541f23 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:48.197968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e257d195-39da-40ff-a227-73a56e541f23 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:48.226578 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1420/2842] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:48 2022] POST /identity/v3/users => generated 292 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:48.231045 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d76d072-d081-497d-bb59-03f8eb779afc admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:48.243002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d76d072-d081-497d-bb59-03f8eb779afc admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:48.248110 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a211f09b-ffcc-45c3-b2bb-01207dfbf0a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:48.248818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a211f09b-ffcc-45c3-b2bb-01207dfbf0a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:48.249105 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a211f09b-ffcc-45c3-b2bb-01207dfbf0a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:48.249401 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a211f09b-ffcc-45c3-b2bb-01207dfbf0a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:48.250088 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a211f09b-ffcc-45c3-b2bb-01207dfbf0a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:48.251775 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a211f09b-ffcc-45c3-b2bb-01207dfbf0a3 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:48.264287 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1423/2843] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:48 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:48.269179 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e257d195-39da-40ff-a227-73a56e541f23 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:48.281386 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e257d195-39da-40ff-a227-73a56e541f23 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:48.287247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7796df6-fe91-4a7c-9a12-0373d3c713c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:48.287878 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7796df6-fe91-4a7c-9a12-0373d3c713c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:48.288154 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7796df6-fe91-4a7c-9a12-0373d3c713c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:48.288469 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7796df6-fe91-4a7c-9a12-0373d3c713c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:48.289171 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7796df6-fe91-4a7c-9a12-0373d3c713c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:48.290871 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7796df6-fe91-4a7c-9a12-0373d3c713c8 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:48.307436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1421/2844] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:48 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:48.312163 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a211f09b-ffcc-45c3-b2bb-01207dfbf0a3 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:48.323931 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a211f09b-ffcc-45c3-b2bb-01207dfbf0a3 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:48.329476 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a06a84-1e39-4e7b-96f6-55beef23ca60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:48.330174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a06a84-1e39-4e7b-96f6-55beef23ca60 admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:48.330456 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a06a84-1e39-4e7b-96f6-55beef23ca60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:48.330762 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a06a84-1e39-4e7b-96f6-55beef23ca60 admin admin] PATH_INFO: `/v3/projects/2839a6ba09bf4f14a39536c82ea7ec53/users/728d6539b44b44a9a9b1efe306087d24/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:48.351388 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a06a84-1e39-4e7b-96f6-55beef23ca60 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2839a6ba09bf4f14a39536c82ea7ec53, user_id=728d6539b44b44a9a9b1efe306087d24, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:48.354250 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a06a84-1e39-4e7b-96f6-55beef23ca60 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:48.371613 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1424/2845] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:48 2022] PUT /identity/v3/projects/2839a6ba09bf4f14a39536c82ea7ec53/users/728d6539b44b44a9a9b1efe306087d24/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:48.389422 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82910138-f38f-48c9-bf39-ec39ebfaa349 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:48.389422 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82910138-f38f-48c9-bf39-ec39ebfaa349 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:48.389615 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82910138-f38f-48c9-bf39-ec39ebfaa349 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:48.421202 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-82910138-f38f-48c9-bf39-ec39ebfaa349 None None] MFA Rules not processed for user `728d6539b44b44a9a9b1efe306087d24`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:48.422418 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-82910138-f38f-48c9-bf39-ec39ebfaa349 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:48.499914 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1422/2846] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:48 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:48.505491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a06a84-1e39-4e7b-96f6-55beef23ca60 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:48.577411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a06a84-1e39-4e7b-96f6-55beef23ca60 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:48.583181 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e504b8-38b7-4cee-9a92-ff0878127e70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:48.583801 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e504b8-38b7-4cee-9a92-ff0878127e70 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:48.584072 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e504b8-38b7-4cee-9a92-ff0878127e70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:48.584384 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e504b8-38b7-4cee-9a92-ff0878127e70 admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:48.585033 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e504b8-38b7-4cee-9a92-ff0878127e70 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:48.586942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e504b8-38b7-4cee-9a92-ff0878127e70 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:48.621599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1425/2847] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:48 2022] POST /identity/v3/projects => generated 383 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:48.625972 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82910138-f38f-48c9-bf39-ec39ebfaa349 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:48.639241 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82910138-f38f-48c9-bf39-ec39ebfaa349 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:48.696075 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:48.712113 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b3d0e5-5637-4a74-b4a9-38b6bdf568f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:48.712829 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b3d0e5-5637-4a74-b4a9-38b6bdf568f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:48.713106 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b3d0e5-5637-4a74-b4a9-38b6bdf568f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:48.713380 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b3d0e5-5637-4a74-b4a9-38b6bdf568f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:48.714018 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b3d0e5-5637-4a74-b4a9-38b6bdf568f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:48.715961 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b3d0e5-5637-4a74-b4a9-38b6bdf568f8 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:48.747738 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1423/2848] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:48 2022] POST /identity/v3/users => generated 292 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:48.749979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e504b8-38b7-4cee-9a92-ff0878127e70 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:48.764561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e504b8-38b7-4cee-9a92-ff0878127e70 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:48.772277 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5484886a-d285-4ac2-a962-a9215b8a50d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:48.773296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5484886a-d285-4ac2-a962-a9215b8a50d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:48.773978 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5484886a-d285-4ac2-a962-a9215b8a50d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:48.774447 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5484886a-d285-4ac2-a962-a9215b8a50d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:48.775843 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5484886a-d285-4ac2-a962-a9215b8a50d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:48.778301 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5484886a-d285-4ac2-a962-a9215b8a50d2 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:48.804386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1426/2849] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:48 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:48.808195 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b3d0e5-5637-4a74-b4a9-38b6bdf568f8 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:48.818841 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b3d0e5-5637-4a74-b4a9-38b6bdf568f8 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:48.825745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b84aa3-99b5-4c9e-96cb-c0d068b319b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:48.826370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b84aa3-99b5-4c9e-96cb-c0d068b319b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:48.826646 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b84aa3-99b5-4c9e-96cb-c0d068b319b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:48.826912 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b84aa3-99b5-4c9e-96cb-c0d068b319b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:48.827568 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b84aa3-99b5-4c9e-96cb-c0d068b319b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:48.829276 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b84aa3-99b5-4c9e-96cb-c0d068b319b3 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:48.844150 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1424/2850] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:48 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:38:48.845416 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5484886a-d285-4ac2-a962-a9215b8a50d2 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:48.858834 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5484886a-d285-4ac2-a962-a9215b8a50d2 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:48.864807 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83acb4fb-3325-416b-be8f-8c6100768f4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:48.865485 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83acb4fb-3325-416b-be8f-8c6100768f4e admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:48.865780 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83acb4fb-3325-416b-be8f-8c6100768f4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:48.866043 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83acb4fb-3325-416b-be8f-8c6100768f4e admin admin] PATH_INFO: `/v3/projects/3503f8f1466a4568b0b58d3400edcf99/users/cdb350f8cf424d989444739a2a4ae96e/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:48.883294 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83acb4fb-3325-416b-be8f-8c6100768f4e admin admin] RBAC: Authorizing `identity:create_grant(project_id=3503f8f1466a4568b0b58d3400edcf99, user_id=cdb350f8cf424d989444739a2a4ae96e, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:48.886038 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83acb4fb-3325-416b-be8f-8c6100768f4e admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:48.901924 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1427/2851] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:48 2022] PUT /identity/v3/projects/3503f8f1466a4568b0b58d3400edcf99/users/cdb350f8cf424d989444739a2a4ae96e/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:48.913334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1c875f-83c3-428d-b9cd-3485204f0101 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:48.913909 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1c875f-83c3-428d-b9cd-3485204f0101 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:48.913909 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1c875f-83c3-428d-b9cd-3485204f0101 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:48.946145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-0b1c875f-83c3-428d-b9cd-3485204f0101 None None] MFA Rules not processed for user `cdb350f8cf424d989444739a2a4ae96e`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:48.947399 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-0b1c875f-83c3-428d-b9cd-3485204f0101 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:49.015605 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1425/2852] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:48 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:49.021324 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83acb4fb-3325-416b-be8f-8c6100768f4e admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:49.027730 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:49.090609 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83acb4fb-3325-416b-be8f-8c6100768f4e admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:49.095460 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f662e1c-209b-4d54-bcfa-6b89a100dfcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:49.096073 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f662e1c-209b-4d54-bcfa-6b89a100dfcd admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:49.096516 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f662e1c-209b-4d54-bcfa-6b89a100dfcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:49.097247 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f662e1c-209b-4d54-bcfa-6b89a100dfcd admin admin] PATH_INFO: `/v3/projects` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:49.097884 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f662e1c-209b-4d54-bcfa-6b89a100dfcd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:49.100059 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f662e1c-209b-4d54-bcfa-6b89a100dfcd admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:49.128649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1428/2853] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:49 2022] POST /identity/v3/projects => generated 385 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:49.132923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1c875f-83c3-428d-b9cd-3485204f0101 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:49.145057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1c875f-83c3-428d-b9cd-3485204f0101 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:49.214640 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc5efad-454f-4317-a682-a8252e7c49e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:49.215429 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc5efad-454f-4317-a682-a8252e7c49e3 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:49.215800 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc5efad-454f-4317-a682-a8252e7c49e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:49.216234 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc5efad-454f-4317-a682-a8252e7c49e3 admin admin] PATH_INFO: `/v3/users` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:49.217137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc5efad-454f-4317-a682-a8252e7c49e3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:49.219441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc5efad-454f-4317-a682-a8252e7c49e3 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:49.245362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1426/2854] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:49 2022] POST /identity/v3/users => generated 299 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:49.249960 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f662e1c-209b-4d54-bcfa-6b89a100dfcd admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:49.263295 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f662e1c-209b-4d54-bcfa-6b89a100dfcd admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:49.268651 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f1fc22-7fe7-4c97-ab89-c13e14f308f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:49.269392 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f1fc22-7fe7-4c97-ab89-c13e14f308f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:49.269781 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f1fc22-7fe7-4c97-ab89-c13e14f308f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:49.269981 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f1fc22-7fe7-4c97-ab89-c13e14f308f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:49.270599 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f1fc22-7fe7-4c97-ab89-c13e14f308f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:49.272279 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f1fc22-7fe7-4c97-ab89-c13e14f308f3 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:49.284932 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1429/2855] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:49 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:49.289664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc5efad-454f-4317-a682-a8252e7c49e3 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:49.302098 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc5efad-454f-4317-a682-a8252e7c49e3 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:49.306976 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc153560-fdf0-4089-8f65-235f37f6cddc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:49.307648 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc153560-fdf0-4089-8f65-235f37f6cddc admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:49.307923 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc153560-fdf0-4089-8f65-235f37f6cddc admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:49.308192 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc153560-fdf0-4089-8f65-235f37f6cddc admin admin] PATH_INFO: `/v3/projects/46d7ac5cfbe046f987ea3d3dfb7eaea0/users/48cec187c9ce4933aa8bc47d2a9a8189/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:49.327286 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc153560-fdf0-4089-8f65-235f37f6cddc admin admin] RBAC: Authorizing `identity:create_grant(project_id=46d7ac5cfbe046f987ea3d3dfb7eaea0, user_id=48cec187c9ce4933aa8bc47d2a9a8189, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:49.330577 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc153560-fdf0-4089-8f65-235f37f6cddc admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:49.347677 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1427/2856] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:49 2022] PUT /identity/v3/projects/46d7ac5cfbe046f987ea3d3dfb7eaea0/users/48cec187c9ce4933aa8bc47d2a9a8189/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:49.352063 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f1fc22-7fe7-4c97-ab89-c13e14f308f3 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:49.363321 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f1fc22-7fe7-4c97-ab89-c13e14f308f3 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:49.429805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1d58b0-71e3-4b50-8411-122801953486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:49.430342 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1d58b0-71e3-4b50-8411-122801953486 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:49.430628 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1d58b0-71e3-4b50-8411-122801953486 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:49.430906 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1d58b0-71e3-4b50-8411-122801953486 admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:49.431563 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1d58b0-71e3-4b50-8411-122801953486 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:49.433234 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1d58b0-71e3-4b50-8411-122801953486 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:49.444804 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1430/2857] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:49 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:49.449026 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc153560-fdf0-4089-8f65-235f37f6cddc admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:49.460864 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc153560-fdf0-4089-8f65-235f37f6cddc admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:49.465830 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102883f3-d105-4de5-beb6-215742e83e33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:49.466515 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102883f3-d105-4de5-beb6-215742e83e33 admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:49.466788 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102883f3-d105-4de5-beb6-215742e83e33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:49.467057 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102883f3-d105-4de5-beb6-215742e83e33 admin admin] PATH_INFO: `/v3/projects/46d7ac5cfbe046f987ea3d3dfb7eaea0/users/48cec187c9ce4933aa8bc47d2a9a8189/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:49.469145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102883f3-d105-4de5-beb6-215742e83e33 admin admin] RBAC: Authorizing `identity:create_grant(project_id=46d7ac5cfbe046f987ea3d3dfb7eaea0, user_id=48cec187c9ce4933aa8bc47d2a9a8189, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:49.472481 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102883f3-d105-4de5-beb6-215742e83e33 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:49.485645 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1428/2858] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:49 2022] PUT /identity/v3/projects/46d7ac5cfbe046f987ea3d3dfb7eaea0/users/48cec187c9ce4933aa8bc47d2a9a8189/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:49.556371 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbdc8b0-6640-422b-9150-e60e23711e42 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:49.556690 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbdc8b0-6640-422b-9150-e60e23711e42 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:49.556970 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbdc8b0-6640-422b-9150-e60e23711e42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:49.588408 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-fbbdc8b0-6640-422b-9150-e60e23711e42 None None] MFA Rules not processed for user `48cec187c9ce4933aa8bc47d2a9a8189`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:49.589649 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-fbbdc8b0-6640-422b-9150-e60e23711e42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:49.660337 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:49.665919 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1431/2859] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:49 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:49.678267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102883f3-d105-4de5-beb6-215742e83e33 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:49.680160 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-102883f3-d105-4de5-beb6-215742e83e33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:49.690296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:49.743690 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102883f3-d105-4de5-beb6-215742e83e33 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:49.747500 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e72f99f-6215-40f1-880b-77203547f50c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:49.748111 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e72f99f-6215-40f1-880b-77203547f50c service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:49.748427 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e72f99f-6215-40f1-880b-77203547f50c service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:49.748708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e72f99f-6215-40f1-880b-77203547f50c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:49.820055 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e72f99f-6215-40f1-880b-77203547f50c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:49.822902 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e72f99f-6215-40f1-880b-77203547f50c service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:49.846147 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1429/2860] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:49 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:38:51.462740 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbdc8b0-6640-422b-9150-e60e23711e42 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:51.480265 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbdc8b0-6640-422b-9150-e60e23711e42 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:51.484945 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca489402-df55-4f74-bcbf-1ee05822dc0c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:51.485564 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca489402-df55-4f74-bcbf-1ee05822dc0c service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:51.485841 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca489402-df55-4f74-bcbf-1ee05822dc0c service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:51.486117 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca489402-df55-4f74-bcbf-1ee05822dc0c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:51.553102 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca489402-df55-4f74-bcbf-1ee05822dc0c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:51.554666 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca489402-df55-4f74-bcbf-1ee05822dc0c service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:51.565778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1432/2861] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:51 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:52.141644 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e72f99f-6215-40f1-880b-77203547f50c service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:52.151460 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e72f99f-6215-40f1-880b-77203547f50c service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:52.155180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8678dcd1-ea81-4d54-ac82-d1dbda5d188a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:52.155787 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8678dcd1-ea81-4d54-ac82-d1dbda5d188a service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:52.156066 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8678dcd1-ea81-4d54-ac82-d1dbda5d188a service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:52.156360 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8678dcd1-ea81-4d54-ac82-d1dbda5d188a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:52.166969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8678dcd1-ea81-4d54-ac82-d1dbda5d188a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:52.168419 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8678dcd1-ea81-4d54-ac82-d1dbda5d188a service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:52.179043 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1430/2862] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:52 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:52.444692 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca489402-df55-4f74-bcbf-1ee05822dc0c service cinder] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:52.446214 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-ca489402-df55-4f74-bcbf-1ee05822dc0c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:52.525842 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca489402-df55-4f74-bcbf-1ee05822dc0c service cinder] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:52.530587 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483139b0-529f-4bda-b78e-083d85175af9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:52.531224 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483139b0-529f-4bda-b78e-083d85175af9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:52.531503 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483139b0-529f-4bda-b78e-083d85175af9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:52.531778 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483139b0-529f-4bda-b78e-083d85175af9 admin admin] PATH_INFO: `/v3/users/9d68420cfdc54d9ba60981894fbbf9dc` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:52.532942 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483139b0-529f-4bda-b78e-083d85175af9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9d68420cfdc54d9ba60981894fbbf9dc)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:52.534927 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483139b0-529f-4bda-b78e-083d85175af9 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:52.592970 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-483139b0-529f-4bda-b78e-083d85175af9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9d68420cfdc54d9ba60981894fbbf9dc'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:52.593666 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-483139b0-529f-4bda-b78e-083d85175af9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9d68420cfdc54d9ba60981894fbbf9dc'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:52.594278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-483139b0-529f-4bda-b78e-083d85175af9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9d68420cfdc54d9ba60981894fbbf9dc'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:52.604149 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-483139b0-529f-4bda-b78e-083d85175af9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9d68420cfdc54d9ba60981894fbbf9dc'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:52.621067 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-483139b0-529f-4bda-b78e-083d85175af9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9d68420cfdc54d9ba60981894fbbf9dc'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:52.636018 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1433/2863] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:52 2022] DELETE /identity/v3/users/9d68420cfdc54d9ba60981894fbbf9dc => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:52.641056 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b083c82d-6704-4246-8e55-5233ce95030f None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:52.641345 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b083c82d-6704-4246-8e55-5233ce95030f None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:52.641624 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b083c82d-6704-4246-8e55-5233ce95030f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:52.663672 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-b083c82d-6704-4246-8e55-5233ce95030f None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:52.664953 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b083c82d-6704-4246-8e55-5233ce95030f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:52.740455 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1431/2864] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 14:38:52.750575 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483139b0-529f-4bda-b78e-083d85175af9 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:52.752184 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-483139b0-529f-4bda-b78e-083d85175af9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:52.782522 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:52.828562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483139b0-529f-4bda-b78e-083d85175af9 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:52.832773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0b3463-3ddc-458b-b4e7-adf5dea9a385 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:52.833508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0b3463-3ddc-458b-b4e7-adf5dea9a385 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:52.833917 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0b3463-3ddc-458b-b4e7-adf5dea9a385 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:52.834292 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0b3463-3ddc-458b-b4e7-adf5dea9a385 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:52.844996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d0b3463-3ddc-458b-b4e7-adf5dea9a385 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:52.846576 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d0b3463-3ddc-458b-b4e7-adf5dea9a385 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:52.857603 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1434/2865] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:53.167770 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b083c82d-6704-4246-8e55-5233ce95030f None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:53.169446 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-b083c82d-6704-4246-8e55-5233ce95030f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:53.183368 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b083c82d-6704-4246-8e55-5233ce95030f None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:53.188816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac83804c-0c2e-4954-871c-74c2ebe7bf81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:53.189657 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac83804c-0c2e-4954-871c-74c2ebe7bf81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:53.190548 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac83804c-0c2e-4954-871c-74c2ebe7bf81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:53.191351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac83804c-0c2e-4954-871c-74c2ebe7bf81 admin admin] PATH_INFO: `/v3/projects/a2dfed22bde5465ba28f70b2eb0293be` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:53.193061 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac83804c-0c2e-4954-871c-74c2ebe7bf81 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a2dfed22bde5465ba28f70b2eb0293be)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:53.195261 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac83804c-0c2e-4954-871c-74c2ebe7bf81 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:53.242918 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:53.247431 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ac83804c-0c2e-4954-871c-74c2ebe7bf81 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a2dfed22bde5465ba28f70b2eb0293be', 'request_id': 'req-ac83804c-0c2e-4954-871c-74c2ebe7bf81'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:53.255133 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ac83804c-0c2e-4954-871c-74c2ebe7bf81 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a2dfed22bde5465ba28f70b2eb0293be', 'request_id': 'req-ac83804c-0c2e-4954-871c-74c2ebe7bf81'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:53.261381 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ac83804c-0c2e-4954-871c-74c2ebe7bf81 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a2dfed22bde5465ba28f70b2eb0293be', 'request_id': 'req-ac83804c-0c2e-4954-871c-74c2ebe7bf81'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:53.274595 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ac83804c-0c2e-4954-871c-74c2ebe7bf81 admin admin] The token cache is being invalidate because project a2dfed22bde5465ba28f70b2eb0293be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:53.274888 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ac83804c-0c2e-4954-871c-74c2ebe7bf81 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:53.275759 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ac83804c-0c2e-4954-871c-74c2ebe7bf81 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:53.277269 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1432/2866] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:53 2022] DELETE /identity/v3/projects/a2dfed22bde5465ba28f70b2eb0293be => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:56.127013 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0b3463-3ddc-458b-b4e7-adf5dea9a385 service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:56.128478 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-7d0b3463-3ddc-458b-b4e7-adf5dea9a385 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:56.141076 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0b3463-3ddc-458b-b4e7-adf5dea9a385 service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:56.198685 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a84930-5e85-4d86-95c0-0481ce78d170 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:56.199319 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a84930-5e85-4d86-95c0-0481ce78d170 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:56.199594 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a84930-5e85-4d86-95c0-0481ce78d170 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:56.199864 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a84930-5e85-4d86-95c0-0481ce78d170 admin admin] PATH_INFO: `/v3/users/728d6539b44b44a9a9b1efe306087d24` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:56.201130 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a84930-5e85-4d86-95c0-0481ce78d170 admin admin] RBAC: Authorizing `identity:delete_user(user_id=728d6539b44b44a9a9b1efe306087d24)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:56.203062 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a84930-5e85-4d86-95c0-0481ce78d170 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:56.254991 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e1a84930-5e85-4d86-95c0-0481ce78d170 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '728d6539b44b44a9a9b1efe306087d24'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:56.255677 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e1a84930-5e85-4d86-95c0-0481ce78d170 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '728d6539b44b44a9a9b1efe306087d24'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:56.256800 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e1a84930-5e85-4d86-95c0-0481ce78d170 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '728d6539b44b44a9a9b1efe306087d24'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:56.265959 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e1a84930-5e85-4d86-95c0-0481ce78d170 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '728d6539b44b44a9a9b1efe306087d24'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:56.280608 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-e1a84930-5e85-4d86-95c0-0481ce78d170 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '728d6539b44b44a9a9b1efe306087d24'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:56.293772 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1435/2867] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:56 2022] DELETE /identity/v3/users/728d6539b44b44a9a9b1efe306087d24 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:38:56.298462 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e48ead-c7c0-4574-8c26-1e2007f12b25 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:56.298947 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e48ead-c7c0-4574-8c26-1e2007f12b25 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:56.299127 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e48ead-c7c0-4574-8c26-1e2007f12b25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:56.320846 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-a3e48ead-c7c0-4574-8c26-1e2007f12b25 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:56.322009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a3e48ead-c7c0-4574-8c26-1e2007f12b25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:56.384089 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1433/2868] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:56.394849 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a84930-5e85-4d86-95c0-0481ce78d170 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:56.396418 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-e1a84930-5e85-4d86-95c0-0481ce78d170 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:56.439357 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:56.455495 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a84930-5e85-4d86-95c0-0481ce78d170 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:56.459853 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd248bae-728b-41e3-bedd-76c5870ac521 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:56.460498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd248bae-728b-41e3-bedd-76c5870ac521 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:56.460777 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd248bae-728b-41e3-bedd-76c5870ac521 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:56.461049 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd248bae-728b-41e3-bedd-76c5870ac521 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:56.470420 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd248bae-728b-41e3-bedd-76c5870ac521 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:56.471891 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd248bae-728b-41e3-bedd-76c5870ac521 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:56.481827 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1436/2869] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:38:56.775937 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e48ead-c7c0-4574-8c26-1e2007f12b25 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:56.777546 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-a3e48ead-c7c0-4574-8c26-1e2007f12b25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:56.791373 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e48ead-c7c0-4574-8c26-1e2007f12b25 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:56.796326 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:56.796964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:56.797243 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:56.797515 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1 admin admin] PATH_INFO: `/v3/projects/2839a6ba09bf4f14a39536c82ea7ec53` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:56.798659 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2839a6ba09bf4f14a39536c82ea7ec53)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:56.800854 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:56.851062 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2839a6ba09bf4f14a39536c82ea7ec53', 'request_id': 'req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:56.857505 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2839a6ba09bf4f14a39536c82ea7ec53', 'request_id': 'req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:56.863861 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2839a6ba09bf4f14a39536c82ea7ec53', 'request_id': 'req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:56.881518 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1 admin admin] The token cache is being invalidate because project 2839a6ba09bf4f14a39536c82ea7ec53 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:56.881814 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:56.882556 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:56.884480 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1434/2870] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:56 2022] DELETE /identity/v3/projects/2839a6ba09bf4f14a39536c82ea7ec53 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:56.888530 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd248bae-728b-41e3-bedd-76c5870ac521 service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:56.899237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd248bae-728b-41e3-bedd-76c5870ac521 service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:56.959321 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c120c82a-dfe7-485d-9f15-4c752ac07f8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:56.959956 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c120c82a-dfe7-485d-9f15-4c752ac07f8c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:56.960237 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c120c82a-dfe7-485d-9f15-4c752ac07f8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:56.960544 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c120c82a-dfe7-485d-9f15-4c752ac07f8c admin admin] PATH_INFO: `/v3/users/cdb350f8cf424d989444739a2a4ae96e` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:56.961679 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c120c82a-dfe7-485d-9f15-4c752ac07f8c admin admin] RBAC: Authorizing `identity:delete_user(user_id=cdb350f8cf424d989444739a2a4ae96e)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:56.963593 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c120c82a-dfe7-485d-9f15-4c752ac07f8c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:57.014570 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c120c82a-dfe7-485d-9f15-4c752ac07f8c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cdb350f8cf424d989444739a2a4ae96e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:57.015316 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c120c82a-dfe7-485d-9f15-4c752ac07f8c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cdb350f8cf424d989444739a2a4ae96e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:57.016113 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c120c82a-dfe7-485d-9f15-4c752ac07f8c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cdb350f8cf424d989444739a2a4ae96e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:57.024065 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c120c82a-dfe7-485d-9f15-4c752ac07f8c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cdb350f8cf424d989444739a2a4ae96e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:57.037381 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-c120c82a-dfe7-485d-9f15-4c752ac07f8c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cdb350f8cf424d989444739a2a4ae96e'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:57.051491 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1437/2871] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:56 2022] DELETE /identity/v3/users/cdb350f8cf424d989444739a2a4ae96e => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:57.345752 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:57.347757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:57.362702 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e50eb8-78a8-4e1a-a044-bbfc98eccca1 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:57.418192 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:57.427016 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:57.427664 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:57.428174 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:57.428658 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae admin admin] PATH_INFO: `/v3/projects/3503f8f1466a4568b0b58d3400edcf99` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:57.430078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae admin admin] RBAC: Authorizing `identity:delete_project(project_id=3503f8f1466a4568b0b58d3400edcf99)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:57.432168 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:57.480928 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3503f8f1466a4568b0b58d3400edcf99', 'request_id': 'req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:57.488127 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3503f8f1466a4568b0b58d3400edcf99', 'request_id': 'req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:57.493947 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3503f8f1466a4568b0b58d3400edcf99', 'request_id': 'req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:57.507688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae admin admin] The token cache is being invalidate because project 3503f8f1466a4568b0b58d3400edcf99 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:57.507981 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:57.508630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:57.510136 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1435/2872] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:57 2022] DELETE /identity/v3/projects/3503f8f1466a4568b0b58d3400edcf99 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:57.514289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c120c82a-dfe7-485d-9f15-4c752ac07f8c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:57.515760 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-c120c82a-dfe7-485d-9f15-4c752ac07f8c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:57.530233 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c120c82a-dfe7-485d-9f15-4c752ac07f8c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:57.590234 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a059fd61-ef1d-43b8-888e-4a43f378734c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:57.590859 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a059fd61-ef1d-43b8-888e-4a43f378734c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:57.591137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a059fd61-ef1d-43b8-888e-4a43f378734c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:57.591409 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a059fd61-ef1d-43b8-888e-4a43f378734c admin admin] PATH_INFO: `/v3/users/48cec187c9ce4933aa8bc47d2a9a8189` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:57.592544 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a059fd61-ef1d-43b8-888e-4a43f378734c admin admin] RBAC: Authorizing `identity:delete_user(user_id=48cec187c9ce4933aa8bc47d2a9a8189)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:57.594454 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a059fd61-ef1d-43b8-888e-4a43f378734c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:57.597501 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:57.645773 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a059fd61-ef1d-43b8-888e-4a43f378734c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '48cec187c9ce4933aa8bc47d2a9a8189'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:57.646436 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a059fd61-ef1d-43b8-888e-4a43f378734c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '48cec187c9ce4933aa8bc47d2a9a8189'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:57.647045 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a059fd61-ef1d-43b8-888e-4a43f378734c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '48cec187c9ce4933aa8bc47d2a9a8189'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:57.655284 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a059fd61-ef1d-43b8-888e-4a43f378734c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '48cec187c9ce4933aa8bc47d2a9a8189'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:57.667601 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a059fd61-ef1d-43b8-888e-4a43f378734c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '48cec187c9ce4933aa8bc47d2a9a8189'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:57.680151 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1438/2873] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:38:57 2022] DELETE /identity/v3/users/48cec187c9ce4933aa8bc47d2a9a8189 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:57.984180 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:57.989767 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:58.005085 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7a34e5-c5c9-46b6-bffa-6d45c0de2eae admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:58.069461 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f015bc-a48b-4789-8bde-642d720e0d7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:58.070178 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f015bc-a48b-4789-8bde-642d720e0d7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:58.070528 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f015bc-a48b-4789-8bde-642d720e0d7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:58.070807 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f015bc-a48b-4789-8bde-642d720e0d7f admin admin] PATH_INFO: `/v3/projects/46d7ac5cfbe046f987ea3d3dfb7eaea0` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:58.071929 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f015bc-a48b-4789-8bde-642d720e0d7f admin admin] RBAC: Authorizing `identity:delete_project(project_id=46d7ac5cfbe046f987ea3d3dfb7eaea0)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:58.073874 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f015bc-a48b-4789-8bde-642d720e0d7f admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:58.122105 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-94f015bc-a48b-4789-8bde-642d720e0d7f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '46d7ac5cfbe046f987ea3d3dfb7eaea0', 'request_id': 'req-94f015bc-a48b-4789-8bde-642d720e0d7f'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:58.128683 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-94f015bc-a48b-4789-8bde-642d720e0d7f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '46d7ac5cfbe046f987ea3d3dfb7eaea0', 'request_id': 'req-94f015bc-a48b-4789-8bde-642d720e0d7f'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:58.137620 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-94f015bc-a48b-4789-8bde-642d720e0d7f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '46d7ac5cfbe046f987ea3d3dfb7eaea0', 'request_id': 'req-94f015bc-a48b-4789-8bde-642d720e0d7f'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:58.153378 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-94f015bc-a48b-4789-8bde-642d720e0d7f admin admin] The token cache is being invalidate because project 46d7ac5cfbe046f987ea3d3dfb7eaea0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:38:58.153692 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-94f015bc-a48b-4789-8bde-642d720e0d7f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:58.154322 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-94f015bc-a48b-4789-8bde-642d720e0d7f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:38:58.155824 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1436/2874] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:38:57 2022] DELETE /identity/v3/projects/46d7ac5cfbe046f987ea3d3dfb7eaea0 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:58.161520 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6b7d43-26cf-49bb-a0e2-5747adac4324 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:58.161836 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6b7d43-26cf-49bb-a0e2-5747adac4324 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:58.162119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6b7d43-26cf-49bb-a0e2-5747adac4324 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:58.182739 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-1a6b7d43-26cf-49bb-a0e2-5747adac4324 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:58.183955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-1a6b7d43-26cf-49bb-a0e2-5747adac4324 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:58.248792 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1439/2875] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:58.257674 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2950c894-82e9-4249-9d87-08d60e2df810 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:58.257899 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2950c894-82e9-4249-9d87-08d60e2df810 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:58.258079 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2950c894-82e9-4249-9d87-08d60e2df810 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:58.278786 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-2950c894-82e9-4249-9d87-08d60e2df810 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:58.280098 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-2950c894-82e9-4249-9d87-08d60e2df810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:58.289197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1437/2876] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:58.293628 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6b7d43-26cf-49bb-a0e2-5747adac4324 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:58.304486 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6b7d43-26cf-49bb-a0e2-5747adac4324 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:58.309131 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c1af05-4108-4ff1-86a0-2fd6ae4e84b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:58.309733 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c1af05-4108-4ff1-86a0-2fd6ae4e84b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:58.310012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c1af05-4108-4ff1-86a0-2fd6ae4e84b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:58.310283 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c1af05-4108-4ff1-86a0-2fd6ae4e84b0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:58.310928 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c1af05-4108-4ff1-86a0-2fd6ae4e84b0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81460) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 14:38:58.311230 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c1af05-4108-4ff1-86a0-2fd6ae4e84b0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:58.312871 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c1af05-4108-4ff1-86a0-2fd6ae4e84b0 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:58.325380 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1440/2877] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Oct 4 14:38:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 14:38:58.330053 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2950c894-82e9-4249-9d87-08d60e2df810 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:58.342142 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2950c894-82e9-4249-9d87-08d60e2df810 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:58.347839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60966a44-0670-4235-a4db-18325b30f350 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:58.348539 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60966a44-0670-4235-a4db-18325b30f350 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:58.348680 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60966a44-0670-4235-a4db-18325b30f350 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:58.348941 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60966a44-0670-4235-a4db-18325b30f350 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:58.349573 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60966a44-0670-4235-a4db-18325b30f350 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:58.351473 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60966a44-0670-4235-a4db-18325b30f350 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:58.357193 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:58.383696 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1438/2878] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:58 2022] POST /identity/v3/projects => generated 383 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:58.384926 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c1af05-4108-4ff1-86a0-2fd6ae4e84b0 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:58.395338 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c1af05-4108-4ff1-86a0-2fd6ae4e84b0 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:58.454810 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648632da-47fe-4c9c-a812-724f545e3be3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:58.455425 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648632da-47fe-4c9c-a812-724f545e3be3 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:58.455754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648632da-47fe-4c9c-a812-724f545e3be3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:58.456125 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648632da-47fe-4c9c-a812-724f545e3be3 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:58.456889 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648632da-47fe-4c9c-a812-724f545e3be3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:58.458867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648632da-47fe-4c9c-a812-724f545e3be3 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:58.475311 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:58.483137 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1441/2879] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:58 2022] POST /identity/v3/users => generated 292 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 14:38:58.487726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60966a44-0670-4235-a4db-18325b30f350 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:58.498351 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60966a44-0670-4235-a4db-18325b30f350 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:58.503820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15adb648-fffa-4c28-a87f-c1c38a756260 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:58.504403 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15adb648-fffa-4c28-a87f-c1c38a756260 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:58.504658 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15adb648-fffa-4c28-a87f-c1c38a756260 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:58.504942 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15adb648-fffa-4c28-a87f-c1c38a756260 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:58.505563 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15adb648-fffa-4c28-a87f-c1c38a756260 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:58.507148 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15adb648-fffa-4c28-a87f-c1c38a756260 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:58.517524 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1439/2880] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:58 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:58.522461 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648632da-47fe-4c9c-a812-724f545e3be3 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:58.533189 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648632da-47fe-4c9c-a812-724f545e3be3 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:58.538234 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c9984d-46d7-491b-9b01-7d0262a8b50d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:58.538840 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c9984d-46d7-491b-9b01-7d0262a8b50d admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:58.539129 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c9984d-46d7-491b-9b01-7d0262a8b50d admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:58.539402 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c9984d-46d7-491b-9b01-7d0262a8b50d admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:58.540046 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c9984d-46d7-491b-9b01-7d0262a8b50d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:58.542481 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c9984d-46d7-491b-9b01-7d0262a8b50d admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:58.559003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1442/2881] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:58 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 14:38:58.563291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15adb648-fffa-4c28-a87f-c1c38a756260 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:58.574135 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15adb648-fffa-4c28-a87f-c1c38a756260 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:58.578845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c34809a-1a13-4cd1-813d-e32e762453ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:58.579482 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c34809a-1a13-4cd1-813d-e32e762453ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:58.579756 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c34809a-1a13-4cd1-813d-e32e762453ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:58.580028 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c34809a-1a13-4cd1-813d-e32e762453ae admin admin] PATH_INFO: `/v3/projects/2749bfd7e216418fb876f0acc1ab3a63/users/aaa38661688b4d3d9ecbeb8ecb820763/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:58.595940 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c34809a-1a13-4cd1-813d-e32e762453ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=2749bfd7e216418fb876f0acc1ab3a63, user_id=aaa38661688b4d3d9ecbeb8ecb820763, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:58.598811 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c34809a-1a13-4cd1-813d-e32e762453ae admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:58.617074 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1440/2882] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:58 2022] PUT /identity/v3/projects/2749bfd7e216418fb876f0acc1ab3a63/users/aaa38661688b4d3d9ecbeb8ecb820763/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:38:58.631258 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4b3380-c05b-48e3-8199-c6f8d81f256e None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:58.631546 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4b3380-c05b-48e3-8199-c6f8d81f256e None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:58.631825 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4b3380-c05b-48e3-8199-c6f8d81f256e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:58.663645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-fc4b3380-c05b-48e3-8199-c6f8d81f256e None None] MFA Rules not processed for user `aaa38661688b4d3d9ecbeb8ecb820763`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:58.664909 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-fc4b3380-c05b-48e3-8199-c6f8d81f256e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:58.734883 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1443/2883] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:58 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:58.740372 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c34809a-1a13-4cd1-813d-e32e762453ae admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:58.806142 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c34809a-1a13-4cd1-813d-e32e762453ae admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:58.811356 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6405e1aa-8e92-4b2f-851e-9c8127df124e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:58.811990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6405e1aa-8e92-4b2f-851e-9c8127df124e admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:58.812268 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6405e1aa-8e92-4b2f-851e-9c8127df124e admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:58.812578 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6405e1aa-8e92-4b2f-851e-9c8127df124e admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:58.813258 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6405e1aa-8e92-4b2f-851e-9c8127df124e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:58.815782 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6405e1aa-8e92-4b2f-851e-9c8127df124e admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:58.845138 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1441/2884] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:58 2022] POST /identity/v3/projects => generated 383 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:58.850174 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4b3380-c05b-48e3-8199-c6f8d81f256e None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:58.862340 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4b3380-c05b-48e3-8199-c6f8d81f256e None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:58.920707 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89f29b4-063a-4e85-9588-678edcdcc489 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:58.921324 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89f29b4-063a-4e85-9588-678edcdcc489 admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:58.921605 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89f29b4-063a-4e85-9588-678edcdcc489 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:58.921881 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89f29b4-063a-4e85-9588-678edcdcc489 admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:58.922526 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89f29b4-063a-4e85-9588-678edcdcc489 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:58.924498 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89f29b4-063a-4e85-9588-678edcdcc489 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:58.945375 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1444/2885] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:58 2022] POST /identity/v3/users => generated 292 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:58.949641 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6405e1aa-8e92-4b2f-851e-9c8127df124e admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:58.961347 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6405e1aa-8e92-4b2f-851e-9c8127df124e admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:58.966039 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e61711-1b7c-4407-a61e-1b7d5fe25dd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:58.966640 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e61711-1b7c-4407-a61e-1b7d5fe25dd7 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:58.966919 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e61711-1b7c-4407-a61e-1b7d5fe25dd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:58.967197 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e61711-1b7c-4407-a61e-1b7d5fe25dd7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:58.967831 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e61711-1b7c-4407-a61e-1b7d5fe25dd7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:58.969460 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e61711-1b7c-4407-a61e-1b7d5fe25dd7 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:58.981078 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1442/2886] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:58.985267 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89f29b4-063a-4e85-9588-678edcdcc489 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:58.995771 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89f29b4-063a-4e85-9588-678edcdcc489 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:59.000759 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ade0e6-1861-4886-8851-60f09af3143c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:59.001379 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ade0e6-1861-4886-8851-60f09af3143c admin admin] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:59.001658 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ade0e6-1861-4886-8851-60f09af3143c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:59.001935 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ade0e6-1861-4886-8851-60f09af3143c admin admin] PATH_INFO: `/v3/roles` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:59.002619 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ade0e6-1861-4886-8851-60f09af3143c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:59.004230 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ade0e6-1861-4886-8851-60f09af3143c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:59.015057 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1445/2887] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:59.019845 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e61711-1b7c-4407-a61e-1b7d5fe25dd7 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:59.030236 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e61711-1b7c-4407-a61e-1b7d5fe25dd7 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:59.035701 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17fed90-5b8c-4b27-bffe-c53682aa98ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:59.036374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17fed90-5b8c-4b27-bffe-c53682aa98ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:59.036660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17fed90-5b8c-4b27-bffe-c53682aa98ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:59.036940 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17fed90-5b8c-4b27-bffe-c53682aa98ca admin admin] PATH_INFO: `/v3/projects/2a7dfcdb757a4362a097b5d87585194b/users/eb7e156f911340b18ae114988ff01ea2/roles/bc6aeaa495ca4f3dbe72905c744c2233` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:59.053326 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17fed90-5b8c-4b27-bffe-c53682aa98ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a7dfcdb757a4362a097b5d87585194b, user_id=eb7e156f911340b18ae114988ff01ea2, role_id=bc6aeaa495ca4f3dbe72905c744c2233)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:59.056009 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17fed90-5b8c-4b27-bffe-c53682aa98ca admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:59.070837 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1443/2888] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:59 2022] PUT /identity/v3/projects/2a7dfcdb757a4362a097b5d87585194b/users/eb7e156f911340b18ae114988ff01ea2/roles/bc6aeaa495ca4f3dbe72905c744c2233 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:59.081355 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9971b3bd-6581-4166-ac6d-ca99550b2166 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:59.081615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9971b3bd-6581-4166-ac6d-ca99550b2166 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:59.081897 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9971b3bd-6581-4166-ac6d-ca99550b2166 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:59.111281 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-9971b3bd-6581-4166-ac6d-ca99550b2166 None None] MFA Rules not processed for user `eb7e156f911340b18ae114988ff01ea2`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:59.113266 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-9971b3bd-6581-4166-ac6d-ca99550b2166 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:59.158509 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:59.181787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1446/2889] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:59 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:59.186556 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17fed90-5b8c-4b27-bffe-c53682aa98ca admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:59.250744 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17fed90-5b8c-4b27-bffe-c53682aa98ca admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:59.256143 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90635f14-f8a6-4b62-ad9b-a5f558860ac3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:59.256863 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90635f14-f8a6-4b62-ad9b-a5f558860ac3 admin admin] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:59.257145 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90635f14-f8a6-4b62-ad9b-a5f558860ac3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:59.257418 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90635f14-f8a6-4b62-ad9b-a5f558860ac3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:59.258094 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90635f14-f8a6-4b62-ad9b-a5f558860ac3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:59.259991 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90635f14-f8a6-4b62-ad9b-a5f558860ac3 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:59.265564 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:59.291072 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1444/2890] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Oct 4 14:38:59 2022] POST /identity/v3/projects => generated 385 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:59.295119 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9971b3bd-6581-4166-ac6d-ca99550b2166 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:59.306976 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9971b3bd-6581-4166-ac6d-ca99550b2166 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:59.364896 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08f896f-a92c-4274-88de-74aab2dbff0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:59.365508 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08f896f-a92c-4274-88de-74aab2dbff0c admin admin] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:59.365792 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08f896f-a92c-4274-88de-74aab2dbff0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:59.366065 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08f896f-a92c-4274-88de-74aab2dbff0c admin admin] PATH_INFO: `/v3/users` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:59.366706 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08f896f-a92c-4274-88de-74aab2dbff0c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:59.368646 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08f896f-a92c-4274-88de-74aab2dbff0c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:59.395742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1447/2891] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Oct 4 14:38:59 2022] POST /identity/v3/users => generated 299 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 14:38:59.397099 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90635f14-f8a6-4b62-ad9b-a5f558860ac3 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:59.407724 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90635f14-f8a6-4b62-ad9b-a5f558860ac3 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:59.413204 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42635af3-db81-4be3-ae07-193a0cdfe60c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:59.413815 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42635af3-db81-4be3-ae07-193a0cdfe60c admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:59.414091 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42635af3-db81-4be3-ae07-193a0cdfe60c admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:59.414362 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42635af3-db81-4be3-ae07-193a0cdfe60c admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:59.414999 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42635af3-db81-4be3-ae07-193a0cdfe60c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:59.416630 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42635af3-db81-4be3-ae07-193a0cdfe60c admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:59.427525 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1445/2892] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:59 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:59.432295 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08f896f-a92c-4274-88de-74aab2dbff0c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:59.444602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08f896f-a92c-4274-88de-74aab2dbff0c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:59.452615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25ca5f5-e4cb-4a46-b36d-574f177b1faa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:59.452615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25ca5f5-e4cb-4a46-b36d-574f177b1faa admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:59.452615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25ca5f5-e4cb-4a46-b36d-574f177b1faa admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:59.452615 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25ca5f5-e4cb-4a46-b36d-574f177b1faa admin admin] PATH_INFO: `/v3/projects/45afbf36dd114cc8a0e0285d56f30836/users/eafc68eff1bb4ac798b41c300373f14b/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:59.470646 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25ca5f5-e4cb-4a46-b36d-574f177b1faa admin admin] RBAC: Authorizing `identity:create_grant(project_id=45afbf36dd114cc8a0e0285d56f30836, user_id=eafc68eff1bb4ac798b41c300373f14b, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:59.473635 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25ca5f5-e4cb-4a46-b36d-574f177b1faa admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:59.487975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1448/2893] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:59 2022] PUT /identity/v3/projects/45afbf36dd114cc8a0e0285d56f30836/users/eafc68eff1bb4ac798b41c300373f14b/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:59.492377 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42635af3-db81-4be3-ae07-193a0cdfe60c admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:59.503277 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42635af3-db81-4be3-ae07-193a0cdfe60c admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:59.571166 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7a269c-a838-49f2-a483-8403339affd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:59.571671 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7a269c-a838-49f2-a483-8403339affd7 admin admin] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:59.571949 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7a269c-a838-49f2-a483-8403339affd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:59.572222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7a269c-a838-49f2-a483-8403339affd7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:59.572890 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7a269c-a838-49f2-a483-8403339affd7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:59.574466 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7a269c-a838-49f2-a483-8403339affd7 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:59.584949 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1446/2894] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Oct 4 14:38:59 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 14:38:59.589445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25ca5f5-e4cb-4a46-b36d-574f177b1faa admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:59.600805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25ca5f5-e4cb-4a46-b36d-574f177b1faa admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:59.605809 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f2d15a-9330-4401-a7da-37b076a47a3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:59.606473 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f2d15a-9330-4401-a7da-37b076a47a3c admin admin] REQUEST_METHOD: `PUT` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:59.606754 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f2d15a-9330-4401-a7da-37b076a47a3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:59.607031 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f2d15a-9330-4401-a7da-37b076a47a3c admin admin] PATH_INFO: `/v3/projects/45afbf36dd114cc8a0e0285d56f30836/users/eafc68eff1bb4ac798b41c300373f14b/roles/7c36efa05ff1457d9f33747e09f2c260` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:59.609148 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f2d15a-9330-4401-a7da-37b076a47a3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=45afbf36dd114cc8a0e0285d56f30836, user_id=eafc68eff1bb4ac798b41c300373f14b, role_id=7c36efa05ff1457d9f33747e09f2c260)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:59.612333 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f2d15a-9330-4401-a7da-37b076a47a3c admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:59.623115 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1449/2895] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Oct 4 14:38:59 2022] PUT /identity/v3/projects/45afbf36dd114cc8a0e0285d56f30836/users/eafc68eff1bb4ac798b41c300373f14b/roles/7c36efa05ff1457d9f33747e09f2c260 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:38:59.633962 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ce37ef-05ff-485b-b04a-11ed86b0c1f5 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:59.634246 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ce37ef-05ff-485b-b04a-11ed86b0c1f5 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:59.634516 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ce37ef-05ff-485b-b04a-11ed86b0c1f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:59.664682 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-00ce37ef-05ff-485b-b04a-11ed86b0c1f5 None None] MFA Rules not processed for user `eafc68eff1bb4ac798b41c300373f14b`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:38:59.665897 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-00ce37ef-05ff-485b-b04a-11ed86b0c1f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:59.739003 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1447/2896] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:38:59 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:38:59.750386 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f2d15a-9330-4401-a7da-37b076a47a3c admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:38:59.752212 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-01f2d15a-9330-4401-a7da-37b076a47a3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:38:59.812057 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f2d15a-9330-4401-a7da-37b076a47a3c admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:38:59.816074 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469318ea-c2f0-4b8d-a75d-693ce6069cc4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:38:59.816715 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469318ea-c2f0-4b8d-a75d-693ce6069cc4 service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:38:59.816997 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469318ea-c2f0-4b8d-a75d-693ce6069cc4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:38:59.817272 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469318ea-c2f0-4b8d-a75d-693ce6069cc4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:38:59.866481 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:38:59.876447 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469318ea-c2f0-4b8d-a75d-693ce6069cc4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:38:59.878787 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469318ea-c2f0-4b8d-a75d-693ce6069cc4 service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:38:59.891219 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1450/2897] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:38:59 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:39:01.424965 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ce37ef-05ff-485b-b04a-11ed86b0c1f5 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:39:01.426526 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-00ce37ef-05ff-485b-b04a-11ed86b0c1f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:39:01.438503 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ce37ef-05ff-485b-b04a-11ed86b0c1f5 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:39:01.442344 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8701a9c-246d-4217-82d6-b2332d49a8c6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:39:01.442947 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8701a9c-246d-4217-82d6-b2332d49a8c6 service cinder] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:39:01.443222 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8701a9c-246d-4217-82d6-b2332d49a8c6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:39:01.443550 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8701a9c-246d-4217-82d6-b2332d49a8c6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:39:01.501745 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:39:01.504631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8701a9c-246d-4217-82d6-b2332d49a8c6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:39:01.506196 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8701a9c-246d-4217-82d6-b2332d49a8c6 service cinder] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:39:01.516631 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1448/2898] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:39:01 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:39:02.074763 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469318ea-c2f0-4b8d-a75d-693ce6069cc4 service cinder] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:39:02.084623 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469318ea-c2f0-4b8d-a75d-693ce6069cc4 service cinder] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:39:02.088398 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ecff3a-fbad-43b5-a4a3-755f5d919d4f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '058c027ade94498fbab7b9347e87c9c3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:39:02.089055 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ecff3a-fbad-43b5-a4a3-755f5d919d4f service cinder] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:39:02.089334 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ecff3a-fbad-43b5-a4a3-755f5d919d4f service cinder] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:39:02.089602 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ecff3a-fbad-43b5-a4a3-755f5d919d4f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:39:02.100569 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ecff3a-fbad-43b5-a4a3-755f5d919d4f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:39:02.102020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ecff3a-fbad-43b5-a4a3-755f5d919d4f service cinder] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:39:02.112988 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1451/2899] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:39:02 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:39:06.064068 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8701a9c-246d-4217-82d6-b2332d49a8c6 service cinder] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:39:06.132370 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8701a9c-246d-4217-82d6-b2332d49a8c6 service cinder] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:39:06.137076 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72612f96-18d4-4cec-9c24-decbfd6b9271 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:39:06.137704 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72612f96-18d4-4cec-9c24-decbfd6b9271 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:39:06.137983 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72612f96-18d4-4cec-9c24-decbfd6b9271 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:39:06.138278 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72612f96-18d4-4cec-9c24-decbfd6b9271 admin admin] PATH_INFO: `/v3/users/aaa38661688b4d3d9ecbeb8ecb820763` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:39:06.139449 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72612f96-18d4-4cec-9c24-decbfd6b9271 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aaa38661688b4d3d9ecbeb8ecb820763)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:39:06.141393 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72612f96-18d4-4cec-9c24-decbfd6b9271 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:39:06.192333 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-72612f96-18d4-4cec-9c24-decbfd6b9271 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aaa38661688b4d3d9ecbeb8ecb820763'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:06.200138 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-72612f96-18d4-4cec-9c24-decbfd6b9271 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aaa38661688b4d3d9ecbeb8ecb820763'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:06.208787 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-72612f96-18d4-4cec-9c24-decbfd6b9271 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aaa38661688b4d3d9ecbeb8ecb820763'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:06.225198 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-72612f96-18d4-4cec-9c24-decbfd6b9271 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aaa38661688b4d3d9ecbeb8ecb820763'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:06.225927 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-72612f96-18d4-4cec-9c24-decbfd6b9271 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aaa38661688b4d3d9ecbeb8ecb820763'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:06.231950 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1449/2900] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:39:06 2022] DELETE /identity/v3/users/aaa38661688b4d3d9ecbeb8ecb820763 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:39:06.236553 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff04e66-44c8-4b9d-b414-a6cc9f1952d3 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:39:06.237188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff04e66-44c8-4b9d-b414-a6cc9f1952d3 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:39:06.237188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff04e66-44c8-4b9d-b414-a6cc9f1952d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:39:06.257551 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-dff04e66-44c8-4b9d-b414-a6cc9f1952d3 None None] MFA Rules not processed for user `4fff265355db498da98d8cb52b62f573`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:39:06.258732 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-dff04e66-44c8-4b9d-b414-a6cc9f1952d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:39:06.325054 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1452/2901] 10.222.0.64 () {58 vars in 958 bytes} [Tue Oct 4 14:39:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:39:06.335904 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72612f96-18d4-4cec-9c24-decbfd6b9271 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:39:06.337385 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-72612f96-18d4-4cec-9c24-decbfd6b9271 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:39:06.399451 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72612f96-18d4-4cec-9c24-decbfd6b9271 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:39:06.403509 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2b2896-1dd6-40c1-a6d5-133bca4fe6e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:39:06.404128 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2b2896-1dd6-40c1-a6d5-133bca4fe6e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:39:06.404441 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2b2896-1dd6-40c1-a6d5-133bca4fe6e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:39:06.404712 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2b2896-1dd6-40c1-a6d5-133bca4fe6e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:39:06.414886 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2b2896-1dd6-40c1-a6d5-133bca4fe6e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:39:06.416330 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2b2896-1dd6-40c1-a6d5-133bca4fe6e9 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:39:06.426156 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1450/2902] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:39:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:39:06.720519 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff04e66-44c8-4b9d-b414-a6cc9f1952d3 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:39:06.722624 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-dff04e66-44c8-4b9d-b414-a6cc9f1952d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:39:06.736113 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff04e66-44c8-4b9d-b414-a6cc9f1952d3 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:39:06.742915 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619f667d-bdcf-488f-aff9-de0a60754940 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:39:06.743658 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619f667d-bdcf-488f-aff9-de0a60754940 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:39:06.743941 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619f667d-bdcf-488f-aff9-de0a60754940 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:39:06.744216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619f667d-bdcf-488f-aff9-de0a60754940 admin admin] PATH_INFO: `/v3/projects/2749bfd7e216418fb876f0acc1ab3a63` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:39:06.745439 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619f667d-bdcf-488f-aff9-de0a60754940 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2749bfd7e216418fb876f0acc1ab3a63)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:39:06.748155 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619f667d-bdcf-488f-aff9-de0a60754940 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:39:06.800955 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-619f667d-bdcf-488f-aff9-de0a60754940 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2749bfd7e216418fb876f0acc1ab3a63', 'request_id': 'req-619f667d-bdcf-488f-aff9-de0a60754940'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:06.807396 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-619f667d-bdcf-488f-aff9-de0a60754940 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2749bfd7e216418fb876f0acc1ab3a63', 'request_id': 'req-619f667d-bdcf-488f-aff9-de0a60754940'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:06.813313 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-619f667d-bdcf-488f-aff9-de0a60754940 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2749bfd7e216418fb876f0acc1ab3a63', 'request_id': 'req-619f667d-bdcf-488f-aff9-de0a60754940'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:06.827086 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-619f667d-bdcf-488f-aff9-de0a60754940 admin admin] The token cache is being invalidate because project 2749bfd7e216418fb876f0acc1ab3a63 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:39:06.827382 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-619f667d-bdcf-488f-aff9-de0a60754940 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:06.828022 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-619f667d-bdcf-488f-aff9-de0a60754940 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:06.829538 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1453/2903] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:39:06 2022] DELETE /identity/v3/projects/2749bfd7e216418fb876f0acc1ab3a63 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:39:06.834532 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2b2896-1dd6-40c1-a6d5-133bca4fe6e9 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:39:06.844897 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2b2896-1dd6-40c1-a6d5-133bca4fe6e9 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:39:06.858501 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:39:06.911253 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7dbf32-c2bf-4c3c-a3e5-a8ea035f3a9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:39:06.911887 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7dbf32-c2bf-4c3c-a3e5-a8ea035f3a9b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:39:06.912164 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7dbf32-c2bf-4c3c-a3e5-a8ea035f3a9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:39:06.912476 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7dbf32-c2bf-4c3c-a3e5-a8ea035f3a9b admin admin] PATH_INFO: `/v3/users/eb7e156f911340b18ae114988ff01ea2` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:39:06.913636 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7dbf32-c2bf-4c3c-a3e5-a8ea035f3a9b admin admin] RBAC: Authorizing `identity:delete_user(user_id=eb7e156f911340b18ae114988ff01ea2)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:39:06.915576 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7dbf32-c2bf-4c3c-a3e5-a8ea035f3a9b admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:39:06.965761 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ef7dbf32-c2bf-4c3c-a3e5-a8ea035f3a9b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eb7e156f911340b18ae114988ff01ea2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:06.973665 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ef7dbf32-c2bf-4c3c-a3e5-a8ea035f3a9b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eb7e156f911340b18ae114988ff01ea2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:06.981194 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ef7dbf32-c2bf-4c3c-a3e5-a8ea035f3a9b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eb7e156f911340b18ae114988ff01ea2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:06.994224 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ef7dbf32-c2bf-4c3c-a3e5-a8ea035f3a9b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eb7e156f911340b18ae114988ff01ea2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:06.994993 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-ef7dbf32-c2bf-4c3c-a3e5-a8ea035f3a9b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eb7e156f911340b18ae114988ff01ea2'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:07.000916 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1451/2904] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:39:06 2022] DELETE /identity/v3/users/eb7e156f911340b18ae114988ff01ea2 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:39:07.287670 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619f667d-bdcf-488f-aff9-de0a60754940 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:39:07.289406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-619f667d-bdcf-488f-aff9-de0a60754940 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:39:07.303026 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619f667d-bdcf-488f-aff9-de0a60754940 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:39:07.361630 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecab3b1-79c0-475d-9575-1a836ad0a7c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:39:07.362281 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecab3b1-79c0-475d-9575-1a836ad0a7c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:39:07.362560 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecab3b1-79c0-475d-9575-1a836ad0a7c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:39:07.362836 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecab3b1-79c0-475d-9575-1a836ad0a7c9 admin admin] PATH_INFO: `/v3/projects/2a7dfcdb757a4362a097b5d87585194b` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:39:07.364020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aecab3b1-79c0-475d-9575-1a836ad0a7c9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a7dfcdb757a4362a097b5d87585194b)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:39:07.365967 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aecab3b1-79c0-475d-9575-1a836ad0a7c9 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:39:07.368587 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:39:07.421994 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-aecab3b1-79c0-475d-9575-1a836ad0a7c9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a7dfcdb757a4362a097b5d87585194b', 'request_id': 'req-aecab3b1-79c0-475d-9575-1a836ad0a7c9'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:07.428676 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-aecab3b1-79c0-475d-9575-1a836ad0a7c9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a7dfcdb757a4362a097b5d87585194b', 'request_id': 'req-aecab3b1-79c0-475d-9575-1a836ad0a7c9'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:07.434522 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-aecab3b1-79c0-475d-9575-1a836ad0a7c9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a7dfcdb757a4362a097b5d87585194b', 'request_id': 'req-aecab3b1-79c0-475d-9575-1a836ad0a7c9'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:07.448742 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-aecab3b1-79c0-475d-9575-1a836ad0a7c9 admin admin] The token cache is being invalidate because project 2a7dfcdb757a4362a097b5d87585194b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:39:07.449040 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-aecab3b1-79c0-475d-9575-1a836ad0a7c9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:07.449719 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-aecab3b1-79c0-475d-9575-1a836ad0a7c9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:07.451255 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1454/2905] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:39:07 2022] DELETE /identity/v3/projects/2a7dfcdb757a4362a097b5d87585194b => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:39:07.456781 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7dbf32-c2bf-4c3c-a3e5-a8ea035f3a9b admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:39:07.458288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-ef7dbf32-c2bf-4c3c-a3e5-a8ea035f3a9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:39:07.472334 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7dbf32-c2bf-4c3c-a3e5-a8ea035f3a9b admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:39:07.534529 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31e94b3-24e1-4c14-ac7d-c3f896d9edfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:39:07.535247 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31e94b3-24e1-4c14-ac7d-c3f896d9edfd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:39:07.535647 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31e94b3-24e1-4c14-ac7d-c3f896d9edfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:39:07.535969 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31e94b3-24e1-4c14-ac7d-c3f896d9edfd admin admin] PATH_INFO: `/v3/users/eafc68eff1bb4ac798b41c300373f14b` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:39:07.537233 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c31e94b3-24e1-4c14-ac7d-c3f896d9edfd admin admin] RBAC: Authorizing `identity:delete_user(user_id=eafc68eff1bb4ac798b41c300373f14b)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:39:07.539287 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c31e94b3-24e1-4c14-ac7d-c3f896d9edfd admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:39:07.582708 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:39:07.589990 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c31e94b3-24e1-4c14-ac7d-c3f896d9edfd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eafc68eff1bb4ac798b41c300373f14b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:07.598670 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c31e94b3-24e1-4c14-ac7d-c3f896d9edfd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eafc68eff1bb4ac798b41c300373f14b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:07.606322 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c31e94b3-24e1-4c14-ac7d-c3f896d9edfd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eafc68eff1bb4ac798b41c300373f14b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:07.619067 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c31e94b3-24e1-4c14-ac7d-c3f896d9edfd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eafc68eff1bb4ac798b41c300373f14b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:07.619747 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c31e94b3-24e1-4c14-ac7d-c3f896d9edfd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eafc68eff1bb4ac798b41c300373f14b'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:07.625499 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1452/2906] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:39:07 2022] DELETE /identity/v3/users/eafc68eff1bb4ac798b41c300373f14b => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:39:07.904980 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecab3b1-79c0-475d-9575-1a836ad0a7c9 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:39:07.906423 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-aecab3b1-79c0-475d-9575-1a836ad0a7c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:39:07.919435 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecab3b1-79c0-475d-9575-1a836ad0a7c9 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:39:07.977370 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:39:07.978005 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:39:07.978286 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:39:07.978563 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65 admin admin] PATH_INFO: `/v3/projects/45afbf36dd114cc8a0e0285d56f30836` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:39:07.979688 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=45afbf36dd114cc8a0e0285d56f30836)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:39:07.981636 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:39:08.032288 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '45afbf36dd114cc8a0e0285d56f30836', 'request_id': 'req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:08.039244 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '45afbf36dd114cc8a0e0285d56f30836', 'request_id': 'req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:08.044910 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '45afbf36dd114cc8a0e0285d56f30836', 'request_id': 'req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:08.057368 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65 admin admin] The token cache is being invalidate because project 45afbf36dd114cc8a0e0285d56f30836 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:39:08.057665 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:08.058313 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:39:08.060020 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1455/2907] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:39:07 2022] DELETE /identity/v3/projects/45afbf36dd114cc8a0e0285d56f30836 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:39:17.622114 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:DELETE=5,keystone:SELECT=14,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:39:18.054252 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=74,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:39:46.861612 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31e94b3-24e1-4c14-ac7d-c3f896d9edfd admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:39:46.863911 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c31e94b3-24e1-4c14-ac7d-c3f896d9edfd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:39:46.928124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31e94b3-24e1-4c14-ac7d-c3f896d9edfd admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:39:46.983284 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214fbd5d-516c-41b3-ae7c-93ec980579f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:39:46.983780 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214fbd5d-516c-41b3-ae7c-93ec980579f7 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:39:46.984053 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214fbd5d-516c-41b3-ae7c-93ec980579f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:39:46.984332 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214fbd5d-516c-41b3-ae7c-93ec980579f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:39:46.986811 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-214fbd5d-516c-41b3-ae7c-93ec980579f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:39:47.085862 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214fbd5d-516c-41b3-ae7c-93ec980579f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:39:47.087667 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214fbd5d-516c-41b3-ae7c-93ec980579f7 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:39:47.103552 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1453/2908] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:39:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 243 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:39:48.291863 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:39:48.293420 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:39:48.307061 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e39cb3-9bbd-4b56-9b18-cb0c48f7af65 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:39:48.310789 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18668e92-7ef1-4639-add8-11ed0c792b42 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:39:48.311445 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18668e92-7ef1-4639-add8-11ed0c792b42 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:39:48.311727 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18668e92-7ef1-4639-add8-11ed0c792b42 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:39:48.312003 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18668e92-7ef1-4639-add8-11ed0c792b42 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:39:48.314519 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-18668e92-7ef1-4639-add8-11ed0c792b42 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:39:48.328893 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18668e92-7ef1-4639-add8-11ed0c792b42 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:39:48.330346 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18668e92-7ef1-4639-add8-11ed0c792b42 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:39:48.342030 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1456/2909] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:39:48 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:39:57.098964 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=74 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:39:58.336800 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=6 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:39:59.305291 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214fbd5d-516c-41b3-ae7c-93ec980579f7 service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:39:59.306715 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-214fbd5d-516c-41b3-ae7c-93ec980579f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:39:59.318281 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214fbd5d-516c-41b3-ae7c-93ec980579f7 service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:39:59.321822 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a0be69-450d-4c64-b8af-580e528bda3c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:39:59.322445 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a0be69-450d-4c64-b8af-580e528bda3c service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:39:59.322757 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a0be69-450d-4c64-b8af-580e528bda3c service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:39:59.323031 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a0be69-450d-4c64-b8af-580e528bda3c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:39:59.324826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-e2a0be69-450d-4c64-b8af-580e528bda3c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:39:59.389939 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a0be69-450d-4c64-b8af-580e528bda3c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:39:59.392128 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a0be69-450d-4c64-b8af-580e528bda3c service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:39:59.403036 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1454/2910] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:39:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:40:05.866011 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18668e92-7ef1-4639-add8-11ed0c792b42 service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:40:05.867561 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-18668e92-7ef1-4639-add8-11ed0c792b42 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:40:05.884985 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18668e92-7ef1-4639-add8-11ed0c792b42 service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:40:05.889646 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc96a1f9-44ca-4d37-b987-465cadedfb93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:40:05.890953 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc96a1f9-44ca-4d37-b987-465cadedfb93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:40:05.891525 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc96a1f9-44ca-4d37-b987-465cadedfb93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:40:05.891805 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc96a1f9-44ca-4d37-b987-465cadedfb93 admin admin] PATH_INFO: `/v3/users/c7028d2ccede4b44b567387bb83c4579` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:40:05.892953 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc96a1f9-44ca-4d37-b987-465cadedfb93 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c7028d2ccede4b44b567387bb83c4579)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:40:05.894867 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc96a1f9-44ca-4d37-b987-465cadedfb93 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:40:05.942818 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-fc96a1f9-44ca-4d37-b987-465cadedfb93 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c7028d2ccede4b44b567387bb83c4579'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:05.943563 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-fc96a1f9-44ca-4d37-b987-465cadedfb93 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c7028d2ccede4b44b567387bb83c4579'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:05.944180 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-fc96a1f9-44ca-4d37-b987-465cadedfb93 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c7028d2ccede4b44b567387bb83c4579'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:05.951697 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-fc96a1f9-44ca-4d37-b987-465cadedfb93 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c7028d2ccede4b44b567387bb83c4579'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:05.964644 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-fc96a1f9-44ca-4d37-b987-465cadedfb93 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c7028d2ccede4b44b567387bb83c4579'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:05.977331 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1457/2911] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:40:05 2022] DELETE /identity/v3/users/c7028d2ccede4b44b567387bb83c4579 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:40:06.150296 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a0be69-450d-4c64-b8af-580e528bda3c service neutron] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:40:06.160783 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a0be69-450d-4c64-b8af-580e528bda3c service neutron] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:40:06.219688 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc85c53-a53a-4452-97da-47516ad28d8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:40:06.220209 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc85c53-a53a-4452-97da-47516ad28d8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:40:06.220513 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc85c53-a53a-4452-97da-47516ad28d8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:40:06.220784 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc85c53-a53a-4452-97da-47516ad28d8f admin admin] PATH_INFO: `/v3/projects/408b16a582484c619ff17416990d5feb` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:40:06.221940 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc85c53-a53a-4452-97da-47516ad28d8f admin admin] RBAC: Authorizing `identity:delete_project(project_id=408b16a582484c619ff17416990d5feb)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:40:06.223839 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc85c53-a53a-4452-97da-47516ad28d8f admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:40:06.270944 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-edc85c53-a53a-4452-97da-47516ad28d8f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '408b16a582484c619ff17416990d5feb', 'request_id': 'req-edc85c53-a53a-4452-97da-47516ad28d8f'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:06.277993 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-edc85c53-a53a-4452-97da-47516ad28d8f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '408b16a582484c619ff17416990d5feb', 'request_id': 'req-edc85c53-a53a-4452-97da-47516ad28d8f'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:06.284039 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-edc85c53-a53a-4452-97da-47516ad28d8f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '408b16a582484c619ff17416990d5feb', 'request_id': 'req-edc85c53-a53a-4452-97da-47516ad28d8f'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:06.295893 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-edc85c53-a53a-4452-97da-47516ad28d8f admin admin] The token cache is being invalidate because project 408b16a582484c619ff17416990d5feb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:40:06.296816 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-edc85c53-a53a-4452-97da-47516ad28d8f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:06.297397 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-edc85c53-a53a-4452-97da-47516ad28d8f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:06.298865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1455/2912] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:40:06 2022] DELETE /identity/v3/projects/408b16a582484c619ff17416990d5feb => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:40:06.302979 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc96a1f9-44ca-4d37-b987-465cadedfb93 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:40:06.304341 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-fc96a1f9-44ca-4d37-b987-465cadedfb93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:40:06.319685 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc96a1f9-44ca-4d37-b987-465cadedfb93 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:40:06.383711 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ed959d-f7f2-4c3a-89b8-c74bd473c33a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:40:06.384406 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ed959d-f7f2-4c3a-89b8-c74bd473c33a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:40:06.384695 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ed959d-f7f2-4c3a-89b8-c74bd473c33a admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:40:06.384975 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ed959d-f7f2-4c3a-89b8-c74bd473c33a admin admin] PATH_INFO: `/v3/users/7f2b7f68137c404396c0e8f71dd6c5e6` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:40:06.386127 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ed959d-f7f2-4c3a-89b8-c74bd473c33a admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f2b7f68137c404396c0e8f71dd6c5e6)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:40:06.388056 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ed959d-f7f2-4c3a-89b8-c74bd473c33a admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:40:06.435719 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-f0ed959d-f7f2-4c3a-89b8-c74bd473c33a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f2b7f68137c404396c0e8f71dd6c5e6'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:06.436410 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-f0ed959d-f7f2-4c3a-89b8-c74bd473c33a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f2b7f68137c404396c0e8f71dd6c5e6'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:06.437801 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-f0ed959d-f7f2-4c3a-89b8-c74bd473c33a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f2b7f68137c404396c0e8f71dd6c5e6'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:06.445499 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-f0ed959d-f7f2-4c3a-89b8-c74bd473c33a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f2b7f68137c404396c0e8f71dd6c5e6'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:06.458369 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-f0ed959d-f7f2-4c3a-89b8-c74bd473c33a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f2b7f68137c404396c0e8f71dd6c5e6'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:06.461054 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:40:06.473951 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1458/2913] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:40:06 2022] DELETE /identity/v3/users/7f2b7f68137c404396c0e8f71dd6c5e6 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:40:06.661569 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc85c53-a53a-4452-97da-47516ad28d8f admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:40:06.663858 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-edc85c53-a53a-4452-97da-47516ad28d8f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:40:06.685910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc85c53-a53a-4452-97da-47516ad28d8f admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:40:06.707882 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:40:06.748137 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be2f4e8-af5d-4319-a6d9-cb0ac7b1999e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:40:06.748714 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be2f4e8-af5d-4319-a6d9-cb0ac7b1999e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:40:06.748989 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be2f4e8-af5d-4319-a6d9-cb0ac7b1999e admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:40:06.749289 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be2f4e8-af5d-4319-a6d9-cb0ac7b1999e admin admin] PATH_INFO: `/v3/projects/1a3633ae850147118ef0ca7e94bd5dc5` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:40:06.750436 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be2f4e8-af5d-4319-a6d9-cb0ac7b1999e admin admin] RBAC: Authorizing `identity:delete_project(project_id=1a3633ae850147118ef0ca7e94bd5dc5)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:40:06.752369 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be2f4e8-af5d-4319-a6d9-cb0ac7b1999e admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:40:06.800283 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8be2f4e8-af5d-4319-a6d9-cb0ac7b1999e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1a3633ae850147118ef0ca7e94bd5dc5', 'request_id': 'req-8be2f4e8-af5d-4319-a6d9-cb0ac7b1999e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:06.806766 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8be2f4e8-af5d-4319-a6d9-cb0ac7b1999e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1a3633ae850147118ef0ca7e94bd5dc5', 'request_id': 'req-8be2f4e8-af5d-4319-a6d9-cb0ac7b1999e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:06.812601 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8be2f4e8-af5d-4319-a6d9-cb0ac7b1999e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1a3633ae850147118ef0ca7e94bd5dc5', 'request_id': 'req-8be2f4e8-af5d-4319-a6d9-cb0ac7b1999e'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:06.827338 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8be2f4e8-af5d-4319-a6d9-cb0ac7b1999e admin admin] The token cache is being invalidate because project 1a3633ae850147118ef0ca7e94bd5dc5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81461) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:40:06.827660 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8be2f4e8-af5d-4319-a6d9-cb0ac7b1999e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:06.828685 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-8be2f4e8-af5d-4319-a6d9-cb0ac7b1999e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:40:06.830400 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1456/2914] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:40:06 2022] DELETE /identity/v3/projects/1a3633ae850147118ef0ca7e94bd5dc5 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:40:16.473157 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=4 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:40:16.824164 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=40,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:40:32.605831 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ed959d-f7f2-4c3a-89b8-c74bd473c33a admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:40:32.607315 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-f0ed959d-f7f2-4c3a-89b8-c74bd473c33a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:40:32.667869 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ed959d-f7f2-4c3a-89b8-c74bd473c33a admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:40:32.717296 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20497cce-088a-4519-b0ed-59612ac4fc0c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:40:32.717907 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20497cce-088a-4519-b0ed-59612ac4fc0c service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:40:32.718216 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20497cce-088a-4519-b0ed-59612ac4fc0c service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:40:32.718490 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20497cce-088a-4519-b0ed-59612ac4fc0c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:40:32.720268 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-20497cce-088a-4519-b0ed-59612ac4fc0c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:40:32.729822 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20497cce-088a-4519-b0ed-59612ac4fc0c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:40:32.731245 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20497cce-088a-4519-b0ed-59612ac4fc0c service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:40:32.740042 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1459/2915] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:40:32 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:40:35.818274 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc4664a-0790-44f4-b277-140ff9661e33 None None] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:40:35.818869 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc4664a-0790-44f4-b277-140ff9661e33 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:40:35.820034 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc4664a-0790-44f4-b277-140ff9661e33 None None] PATH_INFO: `/` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:40:35.821895 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1457/2916] 10.222.0.64 () {54 vars in 921 bytes} [Tue Oct 4 14:40:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:40:35.832753 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e9c76c-d739-4f21-9d84-74f887e23b45 None None] REQUEST_METHOD: `POST` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:40:35.833469 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e9c76c-d739-4f21-9d84-74f887e23b45 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:40:35.833469 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e9c76c-d739-4f21-9d84-74f887e23b45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:40:35.883077 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.auth.core [None req-05e9c76c-d739-4f21-9d84-74f887e23b45 None None] MFA Rules not processed for user `8dbeccfb9a194992959f0b341ef0d4a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81460) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:40:35.884378 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-05e9c76c-d739-4f21-9d84-74f887e23b45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:40:35.891774 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1460/2917] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Oct 4 14:40:35 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:40:35.907847 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc4664a-0790-44f4-b277-140ff9661e33 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:40:35.910717 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-5cc4664a-0790-44f4-b277-140ff9661e33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:40:35.929865 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc4664a-0790-44f4-b277-140ff9661e33 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:40:35.933706 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6f3466-5028-4fd8-8d1c-7387a26a1111 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:40:35.934384 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f3466-5028-4fd8-8d1c-7387a26a1111 service neutron] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:40:35.934729 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f3466-5028-4fd8-8d1c-7387a26a1111 service neutron] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:40:35.935069 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f3466-5028-4fd8-8d1c-7387a26a1111 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:40:35.943980 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6f3466-5028-4fd8-8d1c-7387a26a1111 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:40:35.945565 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6f3466-5028-4fd8-8d1c-7387a26a1111 service neutron] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:40:35.956015 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1458/2918] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:40:35 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:40:36.046002 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8135ab61-0f57-4825-b814-d22029544f98 None None] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:40:36.046922 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8135ab61-0f57-4825-b814-d22029544f98 None None] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:40:36.047340 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8135ab61-0f57-4825-b814-d22029544f98 None None] PATH_INFO: `/` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:40:36.048937 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1461/2919] 10.222.0.64 () {54 vars in 921 bytes} [Tue Oct 4 14:40:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 14:40:36.055534 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06501f3-e97e-4325-a6ff-f0a0ed693aa2 None None] REQUEST_METHOD: `POST` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:40:36.055820 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06501f3-e97e-4325-a6ff-f0a0ed693aa2 None None] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:40:36.056098 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06501f3-e97e-4325-a6ff-f0a0ed693aa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:40:36.101452 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.auth.core [None req-d06501f3-e97e-4325-a6ff-f0a0ed693aa2 None None] MFA Rules not processed for user `03081d7d581745c88c12e9d089fcb2d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81461) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 14:40:36.102681 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d06501f3-e97e-4325-a6ff-f0a0ed693aa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:40:36.179448 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1459/2920] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Oct 4 14:40:36 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 14:40:36.193122 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8135ab61-0f57-4825-b814-d22029544f98 None None] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:40:36.205536 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8135ab61-0f57-4825-b814-d22029544f98 None None] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:40:36.209411 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0683b2a2-809f-4f40-b3ad-5081ee87a996 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dbeccfb9a194992959f0b341ef0d4a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:40:36.210014 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0683b2a2-809f-4f40-b3ad-5081ee87a996 service neutron] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:40:36.210299 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0683b2a2-809f-4f40-b3ad-5081ee87a996 service neutron] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:40:36.210767 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0683b2a2-809f-4f40-b3ad-5081ee87a996 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:40:36.224114 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0683b2a2-809f-4f40-b3ad-5081ee87a996 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:40:36.225996 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0683b2a2-809f-4f40-b3ad-5081ee87a996 service neutron] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:40:36.239779 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1462/2921] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:40:36 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 14:40:36.309993 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06501f3-e97e-4325-a6ff-f0a0ed693aa2 None None] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:40:36.312115 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-d06501f3-e97e-4325-a6ff-f0a0ed693aa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:40:36.401298 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06501f3-e97e-4325-a6ff-f0a0ed693aa2 None None] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:40:36.407968 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c596fed8-d6f3-45ed-92e0-2a08c91a8c8b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '436deb3d60c54a799700986519b2ef68', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:40:36.409002 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c596fed8-d6f3-45ed-92e0-2a08c91a8c8b service placement] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:40:36.409430 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c596fed8-d6f3-45ed-92e0-2a08c91a8c8b service placement] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:40:36.409709 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c596fed8-d6f3-45ed-92e0-2a08c91a8c8b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:40:36.412244 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c596fed8-d6f3-45ed-92e0-2a08c91a8c8b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:40:36.425726 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c596fed8-d6f3-45ed-92e0-2a08c91a8c8b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:40:36.427341 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c596fed8-d6f3-45ed-92e0-2a08c91a8c8b service placement] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:40:36.440341 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1460/2922] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:40:36 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:40:46.233654 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=61 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:40:46.434517 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=75 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:41:57.793234 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0683b2a2-809f-4f40-b3ad-5081ee87a996 service neutron] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:41:57.796648 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-0683b2a2-809f-4f40-b3ad-5081ee87a996 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:41:57.822048 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0683b2a2-809f-4f40-b3ad-5081ee87a996 service neutron] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:41:57.828278 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4459a933-be97-41ed-9981-48b95ce2a56d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '03081d7d581745c88c12e9d089fcb2d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['member', 'service', 'ResellerAdmin', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:41:57.829305 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4459a933-be97-41ed-9981-48b95ce2a56d service nova] REQUEST_METHOD: `GET` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:41:57.829305 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4459a933-be97-41ed-9981-48b95ce2a56d service nova] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:41:57.829664 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4459a933-be97-41ed-9981-48b95ce2a56d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:41:57.832863 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-4459a933-be97-41ed-9981-48b95ce2a56d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:41:57.845052 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4459a933-be97-41ed-9981-48b95ce2a56d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:41:57.847353 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4459a933-be97-41ed-9981-48b95ce2a56d service nova] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:41:57.859918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1463/2923] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:41:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:42:07.855123 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=6 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:42:08.538097 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c596fed8-d6f3-45ed-92e0-2a08c91a8c8b service placement] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:42:08.540044 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c596fed8-d6f3-45ed-92e0-2a08c91a8c8b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:42:08.608000 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c596fed8-d6f3-45ed-92e0-2a08c91a8c8b service placement] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:42:08.614701 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b6c187-4fb7-468b-8161-eb808eb52e73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:42:08.615476 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b6c187-4fb7-468b-8161-eb808eb52e73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:42:08.615753 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b6c187-4fb7-468b-8161-eb808eb52e73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:42:08.616081 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b6c187-4fb7-468b-8161-eb808eb52e73 admin admin] PATH_INFO: `/v3/users/33fb70b5ab4942a38771709eeb3a5789` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:42:08.617267 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b6c187-4fb7-468b-8161-eb808eb52e73 admin admin] RBAC: Authorizing `identity:delete_user(user_id=33fb70b5ab4942a38771709eeb3a5789)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:42:08.619211 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b6c187-4fb7-468b-8161-eb808eb52e73 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:42:08.671420 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c1b6c187-4fb7-468b-8161-eb808eb52e73 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '33fb70b5ab4942a38771709eeb3a5789'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:08.679649 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c1b6c187-4fb7-468b-8161-eb808eb52e73 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '33fb70b5ab4942a38771709eeb3a5789'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:08.686956 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c1b6c187-4fb7-468b-8161-eb808eb52e73 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '33fb70b5ab4942a38771709eeb3a5789'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:08.703124 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c1b6c187-4fb7-468b-8161-eb808eb52e73 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '33fb70b5ab4942a38771709eeb3a5789'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:08.703826 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-c1b6c187-4fb7-468b-8161-eb808eb52e73 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '33fb70b5ab4942a38771709eeb3a5789'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:08.710907 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1461/2924] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:42:08 2022] DELETE /identity/v3/users/33fb70b5ab4942a38771709eeb3a5789 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:42:08.910289 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4459a933-be97-41ed-9981-48b95ce2a56d service nova] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:42:08.929920 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4459a933-be97-41ed-9981-48b95ce2a56d service nova] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:42:09.006091 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85b95b5-29b4-4872-8c04-43749c53dd05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:42:09.006729 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85b95b5-29b4-4872-8c04-43749c53dd05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:42:09.007012 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85b95b5-29b4-4872-8c04-43749c53dd05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:42:09.007297 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85b95b5-29b4-4872-8c04-43749c53dd05 admin admin] PATH_INFO: `/v3/projects/bf0ffb6098014feb89d5d6f61832b03b` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:42:09.008562 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85b95b5-29b4-4872-8c04-43749c53dd05 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bf0ffb6098014feb89d5d6f61832b03b)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:42:09.010466 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85b95b5-29b4-4872-8c04-43749c53dd05 admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:42:09.064645 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d85b95b5-29b4-4872-8c04-43749c53dd05 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bf0ffb6098014feb89d5d6f61832b03b', 'request_id': 'req-d85b95b5-29b4-4872-8c04-43749c53dd05'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:09.072543 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d85b95b5-29b4-4872-8c04-43749c53dd05 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bf0ffb6098014feb89d5d6f61832b03b', 'request_id': 'req-d85b95b5-29b4-4872-8c04-43749c53dd05'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:09.079736 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d85b95b5-29b4-4872-8c04-43749c53dd05 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bf0ffb6098014feb89d5d6f61832b03b', 'request_id': 'req-d85b95b5-29b4-4872-8c04-43749c53dd05'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:09.094261 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d85b95b5-29b4-4872-8c04-43749c53dd05 admin admin] The token cache is being invalidate because project bf0ffb6098014feb89d5d6f61832b03b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:42:09.094549 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d85b95b5-29b4-4872-8c04-43749c53dd05 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:09.095188 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-d85b95b5-29b4-4872-8c04-43749c53dd05 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:09.096712 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1464/2925] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:42:08 2022] DELETE /identity/v3/projects/bf0ffb6098014feb89d5d6f61832b03b => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:42:09.100910 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b6c187-4fb7-468b-8161-eb808eb52e73 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:42:09.102377 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-c1b6c187-4fb7-468b-8161-eb808eb52e73 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:42:09.116062 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b6c187-4fb7-468b-8161-eb808eb52e73 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:42:09.176992 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b853797-435c-401a-9689-ab5fd1b63bd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:42:09.177621 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b853797-435c-401a-9689-ab5fd1b63bd5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:42:09.177895 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b853797-435c-401a-9689-ab5fd1b63bd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:42:09.178163 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b853797-435c-401a-9689-ab5fd1b63bd5 admin admin] PATH_INFO: `/v3/users/8a6d28b8bb5d46c1940cda041d1c56b5` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:42:09.179288 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b853797-435c-401a-9689-ab5fd1b63bd5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8a6d28b8bb5d46c1940cda041d1c56b5)` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:42:09.181268 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b853797-435c-401a-9689-ab5fd1b63bd5 admin admin] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:42:09.203989 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:42:09.232299 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9b853797-435c-401a-9689-ab5fd1b63bd5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8a6d28b8bb5d46c1940cda041d1c56b5'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:09.241050 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9b853797-435c-401a-9689-ab5fd1b63bd5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8a6d28b8bb5d46c1940cda041d1c56b5'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:09.248202 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9b853797-435c-401a-9689-ab5fd1b63bd5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8a6d28b8bb5d46c1940cda041d1c56b5'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:09.260329 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9b853797-435c-401a-9689-ab5fd1b63bd5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8a6d28b8bb5d46c1940cda041d1c56b5'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:09.261058 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.notifications [None req-9b853797-435c-401a-9689-ab5fd1b63bd5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8a6d28b8bb5d46c1940cda041d1c56b5'} {{(pid=81461) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:09.267249 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1462/2926] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Oct 4 14:42:09 2022] DELETE /identity/v3/users/8a6d28b8bb5d46c1940cda041d1c56b5 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 14:42:09.447019 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85b95b5-29b4-4872-8c04-43749c53dd05 admin admin] Authenticating user token {{(pid=81460) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:42:09.448524 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.fernet_utils [None req-d85b95b5-29b4-4872-8c04-43749c53dd05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81460) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:42:09.468302 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85b95b5-29b4-4872-8c04-43749c53dd05 admin admin] Validating token access rules against request {{(pid=81460) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:42:09.546235 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c3c980-4813-44e6-ad1a-4acc6e5a691d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4fff265355db498da98d8cb52b62f573', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '79763f634e4d4d53967547cda84e5ebd', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81460) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:42:09.546913 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c3c980-4813-44e6-ad1a-4acc6e5a691d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:42:09.547200 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c3c980-4813-44e6-ad1a-4acc6e5a691d admin admin] SCRIPT_NAME: `/identity` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:42:09.547481 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c3c980-4813-44e6-ad1a-4acc6e5a691d admin admin] PATH_INFO: `/v3/projects/3b511402d62d48d7ae28d4b36513bacb` {{(pid=81460) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:42:09.548703 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c3c980-4813-44e6-ad1a-4acc6e5a691d admin admin] RBAC: Authorizing `identity:delete_project(project_id=3b511402d62d48d7ae28d4b36513bacb)` {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:42:09.550633 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c3c980-4813-44e6-ad1a-4acc6e5a691d admin admin] RBAC: Authorization granted {{(pid=81460) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:42:09.586973 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:42:09.601415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a4c3c980-4813-44e6-ad1a-4acc6e5a691d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3b511402d62d48d7ae28d4b36513bacb', 'request_id': 'req-a4c3c980-4813-44e6-ad1a-4acc6e5a691d'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:09.608394 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a4c3c980-4813-44e6-ad1a-4acc6e5a691d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3b511402d62d48d7ae28d4b36513bacb', 'request_id': 'req-a4c3c980-4813-44e6-ad1a-4acc6e5a691d'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:09.615286 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a4c3c980-4813-44e6-ad1a-4acc6e5a691d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3b511402d62d48d7ae28d4b36513bacb', 'request_id': 'req-a4c3c980-4813-44e6-ad1a-4acc6e5a691d'} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:09.628918 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a4c3c980-4813-44e6-ad1a-4acc6e5a691d admin admin] The token cache is being invalidate because project 3b511402d62d48d7ae28d4b36513bacb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81460) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 14:42:09.629205 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a4c3c980-4813-44e6-ad1a-4acc6e5a691d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:09.629847 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG keystone.notifications [None req-a4c3c980-4813-44e6-ad1a-4acc6e5a691d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81460) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 14:42:09.631334 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: [pid: 81460|app: 0|req: 1465/2927] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Oct 4 14:42:09 2022] DELETE /identity/v3/projects/3b511402d62d48d7ae28d4b36513bacb => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 14:42:19.263722 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:DELETE=7,keystone:SELECT=20,keystone:INSERT=1 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:42:19.625415 dv-cinder-860260-1-iscsi devstack@keystone.service[81460]: DEBUG dbcounter [-] [81460] Writing DB stats keystone:SELECT=12,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81460) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 14:42:51.050629 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b853797-435c-401a-9689-ab5fd1b63bd5 admin admin] Authenticating user token {{(pid=81461) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 14:42:51.052051 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-9b853797-435c-401a-9689-ab5fd1b63bd5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:42:51.122374 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b853797-435c-401a-9689-ab5fd1b63bd5 admin admin] Validating token access rules against request {{(pid=81461) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 14:42:51.183868 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82305db6-c1e2-47ef-b140-27de021d3451 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '436deb3d60c54a799700986519b2ef68', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '895ece8b70ef44adadc3416f81a80731', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81461) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 14:42:51.184423 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82305db6-c1e2-47ef-b140-27de021d3451 service placement] REQUEST_METHOD: `GET` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 14:42:51.184705 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82305db6-c1e2-47ef-b140-27de021d3451 service placement] SCRIPT_NAME: `/identity` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 14:42:51.184977 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82305db6-c1e2-47ef-b140-27de021d3451 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81461) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 14:42:51.186713 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.fernet_utils [None req-82305db6-c1e2-47ef-b140-27de021d3451 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81461) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 14:42:51.199250 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82305db6-c1e2-47ef-b140-27de021d3451 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 14:42:51.200846 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82305db6-c1e2-47ef-b140-27de021d3451 service placement] RBAC: Authorization granted {{(pid=81461) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 14:42:51.223540 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: [pid: 81461|app: 0|req: 1463/2928] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Oct 4 14:42:51 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 14:43:01.215285 dv-cinder-860260-1-iscsi devstack@keystone.service[81461]: DEBUG dbcounter [-] [81461] Writing DB stats keystone:SELECT=58 {{(pid=81461) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}